Coding and Dismantling Stuff

Don't thank me, it's what I do.

About the author

Russell is a .Net developer based in Lancashire in the UK.  His day job is as a C# developer for the UK's largest online white-goods retailer, DRL Limited.

His weekend job entails alternately demolishing and constructing various bits of his home, much to the distress of his fiance Kelly, 3-year-old daughter Amelie, and menagerie of pets.

TextBox

  1. Fix dodgy keywords Google is scraping from my blog
  2. Complete migration of NHaml from Google Code to GitHub
  3. ReTelnet Mock Telnet Server à la Jetty
  4. Learn to use Git
  5. Complete beta release FHEMDotNet
  6. Publish FHEMDotNet on Google Code
  7. Learn NancyFX library
  8. Pull RussPAll/NHaml into NHaml/NHaml
  9. Open Source Blackberry Twitter app
  10. Other stuff

Enterprise Logging Part 2 - Logging From .Net to RabbitMq via Log4Net

Hi all,

I'm back with Part 2 of my plan to log, search, graph and alert on milllions of log messages with Log4Net, RabbitMq, GrayLog2 and NEsper. I must confess, I've been pretty astonished at the reaction to part 1 (a week later I was still seeing 4 times my normal blog traffic), so I guess I've gotta follow through and see the plan to fruition!

I'm going to start at the application end today, I'm going to install RabbitMq on a Windows box, and demonstrate a .Net application sending messages to it via a custom Log4Net appender (Gelf4Net over at http://github.com/RussPAll/gelf4net).

Installing RabbitMq

First off, let's get RabbitMq up and running. Before we go ahead and install it, we'll need to install the Erlang tools, as follows:

 

Erlang OTP Installer

  1. Navigate to the Erlang Download page (http://www.erlang.org/download.html)
  2. Download and run the Windows binary file for your operating system (there are 32 bit and 64 bit builds)
  3. Click through the installer, the default options all seemed sensibile to me.
  4. I'd love to be able to tell you how to test your Erlang install, but I spent a few minutes coding with the Erlang Windows environment and didn't get very far, must try harder next time!

Now that we've installed Erlang, the process for RabbitMq is pretty much the same:

 

RabbitMq Installer

  1. Navigate to the RabbitMQ Download page (http://www.rabbitmq.com/download.html)
  2. Download and run the Windows installer (rabbitmq-server-2.8.4.exe)
  3. Again click through the installer using the defaults.
  4. After the install is finished, to check your install, open the RabbitMq command prompt (Start menu -> All Programs -> RabbitMq Server)
  5. Enter the command "rabbitmqctl.bat status", you should see a JSON object printed to the console showing things such as the process ID, the Erlang version, etc.

When I first did this two weeks back now, I was using the command line tools for everything – configuring, checking queues, adding and removing exchanges (more on that another time), it turns out I was missing the awesomeness that is...

The RabbitMq Management Web GUI

It turns out there's a web-based Management Plugin. I would recommend using this straightaway. The tool is actually bundled with the RabbitMq installer; it just needs activating by running the following commands in sequence from the RabbitMq command prompt:

rabbitmq-plugins enable rabbitmq_management
rabbitmq-service.bat stop
rabbitmq-service.bat install
rabbitmq-service.bat start

You should now be able to open up your Management GUI by browsing to http://localhost:55672/ and entering login details "guest" and "guest".

 

RabbitMQ Web-Based Management Console

A Simple Console Log4Net Example

Now that our RabbitMq infrastructure is up and running, let's turn our attention to Log4Net. For those of you uninitiated, Log4Net is a widely used logging framework for the.Net platform, maintained by the Apache Software Foundation folks. As well as being one of the most widely used .Net logging tools available, it supports an “appender” plugin system, allowing you to get logs out of your app to pretty much any external storage you can imagine (Twitter, IRC, MSMQ, SMS, etc).

If you're au-fait with Log4Net, feel free to download the below source code and skip to the next section. Otherwise, I'd follow along with the below so that it sticks.


Log4NetConsoleDemoApp.zip (111.79 kb)

A Simple Console Log4Net Example - Source Code

 

 

Visual Studio - Creating a New Console Application

  1. Let's spin up visual studio and create a new console application project.
  2. Right click the new project in your Solution Explorer, and select Manage Nuget Packages
  3. Click the “Online” tab on the left of the Nuget window, then search for and install Log4Net.
  4. After Log4Net finishes installing, we'll need to configure it. This is a two part process – first we'll need to add the following to a new app.config file (when we come to configure the Gelf4Net appender we'll come back to this configuration again in a little more detail):
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <configSections>
        <section name="log4net"
          type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
      </configSections>
     
      <log4net>
        <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender" >
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
          </layout>
        </appender>
        <root>
          <level value="INFO" />
          <appender-ref ref="ConsoleAppender" />
        </root>
      </log4net>
    </configuration>
  5. Next we need to tell our application to read this configuration, by adding the following line to your program.CS before the line starting "namespace".
    [assembly: log4net.Config.XmlConfigurator(Watch = true)]
  6. Finally, to invoke Log4Net, let's add some logging statements to our application. Add the following code to the main function in program.cs
    var logger = log4net.LogManager.GetLogger("");
    logger.Debug("This will not be output");
    logger.Info("This message will be logged");
    logger.Warn("This warn message be logged too");
    logger.Error("And this one too, normally accompanied by an exception", new Exception());
    logger.Fatal("Goodbye cruel world...");
    Console.ReadLine();

If you now execute your console application, you should see the following output to your screen.

Introducing Gelf4Net - Our AMQP-Savvy Log4Net Appender

Now that we've seen RabbitMq and Log4Net, let's wire the two together. Remember I mentioned earlier about extending Log4Net using custom appenders? It turns out that someone's already on with this - a quick Google turned up Gelf4Net (http://github.com/jjchiw/gelf4net) – but the AMQP support in this appender is untested. Thanks to the wonders of GitHub, I've been in, tested it and fixed a few bugs, and submitted a pull request back to the original author. In the meantime, my fork's over at http://github.com/RussPAll/gelf4net, and you can download my fixed DLL from http://github.com/RussPAll/gelf4net/downloads .

Let's get our console application to start logging to RabbitMq:

  1. Download the Gelf4Net DLL and place it in a suitable folder in our solution
  2. In Visual Studio, right-click our console project and add a reference to the DLL.
  3. Using Nuget Package Manager, we need to add references to Json.NET and RabbitMQ.Client.
  4. In your app.config file, add a new appender tag as below
    <appender name="GelfAppender" type="Esilog.Gelf4net.Appender.Gelf4NetAppender, Esilog.Gelf4net">
      <param name="GrayLogServerHostIpAddress" value="localhost" />
      <param name="Facility" value="Gelf4NetConsoleTest" />
      <param name="AdditionalFields" value="app:RandomSentece,version:1.0" />
      <param name="IncludeLocationInformation" value="true"/>
      <param name="GrayLogServerAmqpUser" value="guest" />
      <param name="GrayLogServerAmqpPassword" value="guest" />
      <param name="GrayLogServerAmqpQueue" value="Queue1" />
      <param name="GrayLogServerAmqpVirtualHost" value="/" />
      <param name="UseUdpTransport" value="false" />
      <layout type="log4net.Layout.PatternLayout">
         <param name="ConversionPattern" value="%-5p%d{yyyy-MM-dd hh:mm:ss} – %m%n"/>
      </layout>
    </appender>
  5. Add a reference to the new appender to your root logger, as follows:
    <root>
      <level value="INFO" />
      <appender-ref ref="ConsoleAppender" />
      <appender-ref ref="GelfAppender" />
    </root>

If you now spin up your console application, fingers crossed you'll get the exact same console output you saw a moment ago. Now let's run up the RabbitMQ management console, click the "Queues" tab, and hey presto, you should now have four messages sat waiting in the Queue1 queue.

Source Code For Download


Log4NetAmqpApp.zip (414kb)

A Simple AMQP / Gelf Log4Net Example - Source Code


Categories: Architecture | Hacking
Permalink | Comments (2)

Comments (2) -

Joe Hosman United States

01 August 2012 16:15

Joe Hosman

I'm ready for part 3! Smile

Darius Lithuania

29 September 2012 14:08

Darius

Ah, so the most interesting part about NEsper was never written Smile.  

I would love to hear how did it go in the end

Pingbacks and trackbacks (1)+

Add comment

  Country flag

biuquote
  • Comment
  • Preview
Loading