Overview

Logging has always been an afterthought for too many organizations. The ILogger interface brings a robust set of logging to your application. What makes this special isn't the mere ability to log, but rather the content of what gets logged, the configurability, and the performance. Most loggers are synchronous, meaning that your application has to wait while the logging occurs. In this implementation, logging is asynchronous (the retrieval of some data may be synchronous).

Here are the basic steps for how something gets logged:

Logging statement appears in your code (see Statements)
Precheck occurs to determine if logging will actually occur for this call (see Precheck)
The logger will initiate an object containing all the information to be logged (see Information)
The logger will kick off an asynchronous thread to do the actual logging (see Asynchronous)
Additional properties on the logging object are set (see Additional Properties)
Logging object is sent to all enabled loggers (see Loggers)

If your logging call is for something that will run a timer (eg. API, Web Service, Database, etc), the logging object will be returned to the caller. When the action has completed, this object is passed to a completion logging statement where the following occurs:

Ensure a valid logging object was given (if this object is null, nothing will be logged)
Logger will capture any additional information to be logged (see Update Information)
Stops the timer that was started in the initial logging statement
The logger will kick off an asynchronous thread to the the rest of the work (see Asynchronous)
New thread will wait for the initial logging statement to complete (see Previous Completion)
Additional properties on the logging object are set (see Additional Properties)
Alerts will be checked and possibly triggered
Logging object is sent to all enabled loggers (see Loggers)

Lastly:

Any exceptions that occurred during logging will be captured via Email (see Exception Handling)
You can customize what gets logged (see Customizations)
There are many configurations via this implementation using ILoggers, ILoggerInformation, and ILoggerConfiguration

Interface: ILogger

Mock: NonLogger

Implementation: Logger

Statements

The ILogger lists all the possible calls that you can make to the logger. These are broken into 2 categories:

Insert-only statements
Updateable statements

Insert-Only

All of these logging methods will log what needs to be logged (no subsequent logging statements can be made with the logging objects).

Method	Information Class	Where Used	Description
Log	MessageInformation	HttpDataMiddleware: (Warning) If a non-secure request is received when HTTPS is required SignalR: (Suspend) On SignalR communication issues	Logs a simple user-defined message
Unauthorized	UnauthorizedInformation	AuthenticationMiddleware: Unauthenticated or Unauthorized request	Logs that an unauthorized call was made in the system (eg. HTTP 401/403)
Exception	ExceptionInformation	ExceptionMiddleware	Records that an unhandled (fatal) exception occurred in the system
HiddenException	ExceptionInformation	SafeTry.LogException	Records that an exception occurred in the system that was non-fatal
Custom1	CustomInformation1		Allows you to create your own design-time class (see Customizations)
Custom2	CustomInformation2		Allows you to create your own design-time class (see Customizations)
Custom3	CustomInformation3		Allows you to create your own design-time class (see Customizations)
Generic	BaseLogInformation		Allows you to create your own generic design-time class (see Customizations)

Updateable

The following methods will be called at the start of something, then again when something ends. The TimerBaseInformation object will be returned from that "entry" call, and this object should be passed to the "exit" method.

Entry Method	Exit Method	Information Class	Where Used	Description
AttributeEntry	AttributeExit	AttributeInformation	None - there are some Attributes, but none that are logging anything	Logs the execution of an attribute
ApiEntry	ApiExit	ApiInformation	ApiLoggingMiddleware	Logs the execution of the API endpoint
DatabaseEntry	DatabaseExit	DatabaseInformation	Repository Calls Transactions	Logs the execution of a database call
WebServiceEntry	WebServiceExit	WebServiceInformation	Web Services: The actual call, not the preparation, nor parsing of result	Logs the execution of a web service call
MiddlewareEntry	MiddlewareExit	MiddlewareInformation	ApiLoggingMiddleware AuthenticationMiddleware ExceptionMiddleware JwtMiddleware	Logs the execution of a middleware call in the API Pipeline
TimerStart	TimerEnd	TimerBaseInformation		Allows you to create ad-hoc timer classes in your code for tracking how long sections of code run (see Customizations)
CustomTimer1Start	CustomTimerEnd	CustomTimerInformation1		Allows you to create your own design-time class which acts as a timer (see Customizations)
CustomTimer2Start	CustomTimerEnd	CustomTimerInformation2		Allows you to create your own design-time class which acts as a timer (see Customizations)
CustomTimer3Start	CustomTimerEnd	CustomTimerInformation3		Allows you to create your own design-time class which acts as a timer (see Customizations)
Entry	Exit	User-supplied TimerBaseInformation		If you don't want to use the pre-built classes inheriting from TimerBaseInformation, you can create your own customized object (see Customizations)

Precheck

The pre-check step is a performance enhancement to quickly return if the "Type" (see customizations of object being logged is not enabled. The ILoggerConfiguration (see configurations) has 2 properties that drive this:

Enabled
LoggerEnabled()

If both of these return true, then the precheck passes. If either of these return false, the precheck fails and the logging statement will conclude without performing any data lookups or actual logging.

Information

The information that gets logged is critical, and is a huge reason for the complexity of this logging sytem. All things that will be logged will contain the following pieces of information (see BaseLogInformation):

Property	Description	Customizeable
Identifier	A unique identifier (Guid) for each log statement. This identifier will be used to link the 'insert' statement with the 'update' statement.
Timestamp	When the logging statement was called
Severity	Similar to logging level, this gives an indication of how critical the message is. See TraceEventType for possible values
Order	All statements within a Request will be ordered. You can sort by this column to see all the statements for a given request in the correct order. Any calls to another web service will also be ordered as part of this. Eg. A, B, BA, BB, C: Three statements were logged on the main request, and the 2nd "B" request was a web service call that logged 2 statements.	LogOrderPrefix LoggingOrder
RequestId	A unique identifier (Guid) that is used for all logging statements within a single Http Request	RequestId
Session	If your front end application is generating a sessionId, it can pass this along to be logged. It should pass this in the Http header: SessionIdHeader. This will tie all requests together for a given "session" (eg. multiple Http Requests using the same window without refresh)	SessionId
UserId	If the user is logged in, this is their Id
IdentityProperties	If you have custom items in your identity object that will always need to be logged, you should: Create your own custom class that inherits from BaseIdentity Add in your custom properties to that class Override the GetCustomLoggingProperties() method and return any properties in that collection In your logger, ensure IncludeIdentity is enabled Retrieve that property from the dictionary
Message	The main message to be logged
Category	A category to group messages. For Timer statements (eg. has Entry() and Exit() methods), the Category will match the Type
Machine	Name of the physical machine this code was running on
Application	Name of the application (eg. The same application could be running on multiple machines for load balancing)
StaticProperties	Any additional properties you want included: Derive and implement StaticProperties Update service registration (startup.cs) with your new class In your logger, ensure IncludeStatic is enabled Retrieve that property from the dictionary

Most of these properties get set immediately in the context of the active application thread. However, the following properties are set later since they do not depend on an active object on the main thread:

There is a slight overhead to setting all of these properties, but they are small and have been optimized. Any other custom properties should be set as follows:

Constructor: Never
Initialize(): This is what gets called to save off any information that was passed into the logging call. You should avoid directly setting custom properties in this method unless it is not thread-safe. You should instead just save off the object(s) into a member variable(s).
SetProperties(): This is where you'll take the saved object(s) from Initialize() and actually set the real properties (see Additional Properties)

Here are all of the custom properties for each logging type. Note that the "key" is provided, and can be used to "hide" the value (see configurations for more information around these).

Type	Key	When Set	Description
API	Route	Initialize	Unique route (template)
API	Http Method	Initialize	Request Method (eg. POST/PUT/GET/ETC)
API	Http Headers	Initialize (Formatting done in SetProperties)	The entire collection of Http Request headers
API	Query String	Initialize (Formatting done in SetProperties)	The querystring in the URL
API	Form Data	Initialize (Formatting done in SetProperties)	Form data in the Http Request
API	Request Data	Initialize (Formatting done in SetProperties)	Data in the Http Request that was posted (or other method)
API	Browser	Initialize	Type of user browser (Eg. "User-Agent" header)
API	Http Response	External Call	Http Response Status Code (Eg. 200)
API	Http Result	External Call	The complete Http Response object
Attribute	User-Supplied	Initialize	Any attribute information to be saved
Database	Connection	Initialize	Name of the database connection repo
Database	SQL Parameters	SetProperties	SQL Parameters as name/value pairs
Exception, Hidden Exception	Stack Trace	SetProperties	The stack trace of the exception
Exception, Hidden Exception	Source	SetProperties	Source of the exception
Exception, Hidden Exception	Title	Initialize	Type/Title for the exception
Web Service	Route	Initialize	Unique route for the web service call
Web Service	Http Method	Initialize	Request Method (eg. POST/PUT/GET/ETC)
Web Service	Request Data	SetProperties	Data in the Http Request that was posted (or other method)
Web Service	Http Response	External Call	Http Response Status Code (Eg. 200)
Web Service	Http Result	External Call	The complete Http Response object

Asynchronous

The Logger implementation of the ILogger interface runs asynchronously. The logging statement, precheck, and setting of information runs synchronously. But the remainder of the call will be done asynchronously (in the background, no await). Eg. SetProperties() and loggers. Even when you await the completion of the statement, this will continue to execute asynchronously. It is because of this asynchronous nature that you should be careful what gets set in Initialize() vs SetProperties() methods. Eg. the HttpRequest could be gone by the time the logger is used to actually log information related to this, so anything that relies on the HttpRequest object (or anything else that isn't thread-safe) should go in the Initialize() method. If you can find a way to push this into the SetProperties() method, or at least do the majority of processing in that method, then your overall logging performance will increase since those items will run asynchronously and not cause the main thread to wait during Initialize().

Additional Properties

Any custom properties on your logging information class will need to do the following:

Following setting information, there may still be some custom properties that need to be set
Obfuscation of the values based on HideKeys() (see ILoggerConfiguration)
Taking the property, and assigning it to a Dictionary(string, string) - eg. key/value

For non-timer statements, and for all entry() methods, SetProperties() is where this will all be done. The 3 dictionaries (HighProperties, MedProperties, LowProperties) are used by the loggers. Eg. some loggers might only save the 'High' properties.

For update timer statements (eg. exit() methods), UpdateProperties() is where this will all be done. Before this method is called, the 3 dictionaries will be cleared out so that only updated items will be logged on the update.

Type	Key	Insert	Update	Dictionary
API	Route	Yes		High
API	Http Method	Yes		High
API	Http Headers	Yes		Med
API	Query String	Yes		Med
API	Form Data	Yes		Med
API	Request Data	Yes		Med
API	Browser	Yes		Low
API	Http Response		Yes	High
API	Http Result		Yes	Med
Attribute	User-Supplied	Yes		Med
Database	Connection	Yes		High
Database	SQL Parameters	Yes		Med
Exception, Hidden Exception	Stack Trace	Yes		High
Exception, Hidden Exception	Source	Yes		Med
Exception, Hidden Exception	Title	Yes		Low
Web Service	Route	Yes		High
Web Service	Http Method	Yes		High
Web Service	Request Data	Yes		Med
Web Service	Http Response		Yes	High
Web Service	Http Result		Yes	Med

Loggers

A logger is a how the information gets transferred to a physical storage location. Eg. File, Database, etc. A logger has the following characteristics:

Class inherits from BaseLogger
Is registered in your implementation of ILoggers
Is configured to perform logging - LoggerEnabled() method

Available loggers (you can create your own too):

Type	Name	Description	Recommended Types	Default Dictionaries
AwsLogger	AWS	Sends to CloudWatch (Not currently an option... You can view source code for example implementation)	All	All
AzureLogger	Azure	Sends to EventHub (Not currently an option... You can view source code for example implementation)	All	All
DebugLogger	Debug	Debugging messages in visual studio utilizing Debug.WriteLine()	Exception;Hidden Exception;HTTP Response Error;Long Running;Message;Timer	High
EmailLogger	Email	Sends an Email to someone	Identity;Static;High;Med	Exception;Hidden Exception
FileLogger	File	Writes to a file. A single file will be created every day in the format of $"log_{dt.DayOfYear}.log". File location is given by the variable "Logging_FilePath" - and put into subfolder of "Logs" at that path. If the severity of the thing being logged is Warning, Error, or Critical, then a 2nd logfile will be created. Location of this log file will be in a subfolder of the Logs directory called "Highlights" with a naming convention of $"log_highlights_{dt.Year}_{dt:MMM}.log".	Identity;Static;High;Med	API;Exception;Hidden Exception;HTTP Response Error;Long Running;Message;Timer;Web Service
Log4NetLogger	Log4Net	Sends to Log4Net which has it's own logging configurations	All	All

Registration of which loggers the system knows about is done via the ILoggers interface. The default registration is a Transient which returns an empty list (no loggers). Your implementation registration may wish to be Singleton (especially if there are multiple injections or any kind of business logic/lookups).

Update Information

This only impacts 2 calls:

ApiExit()
WebServiceExit()

In both cases, a call to SaveResult() will occur which will save off the returned Status Code and Response body. These will be the only candidates to be logged on the update statement (along with time to execute).

Previous Completion

Because logging is mostly asynchronous, the Exit() call might happen before the Entry() call has completed. Depending on your logger (eg. Database), you may need to capture the generated LogID (to be placed in NotResetProperties). The Exit() log statement will depend on this completing, and it's good practice to ensure the Entry() log statement appears 1st in the logs. Therefore, the Exit() call will wait for the Entry() log statement to complete. It does this by setting the SetComplete flag when the Entry() statment completes. The Exit() statement will await the setting of this flag (technically sleep for 100ms and recheck for up to 1 minute).

Exception Handling

Everything that occurs within the asynchronous portion of execution will have all exceptions handled. Because we are already in the logger, it doesn't make sense to log the exception (possible recursive loop). Instead, the logging error will be e-mailed and any exceptions during the Email process will be ignored. Note this utilizes the Email type: HiddenException.

Configurations

The ILoggerConfiguration contains a number for configurations. The main implementation is an abstract class LoggerConfiguration. The registered implementation is SphyrnidaeLoggerConfiguration. You will likely want to replace this with a different implementation. For best performance, it is recommended to register your implementation as Scoped. You can then save off the current configuration for all log statements in the request.

Property/Method	Description	Default Implementation
Enabled	Global flag that allows you to completely disable logging. If this is true, a lot more checks might still prevent statements from being logged.	True
TypeEnabled()	The Type of the thing being logged can be enabled/disabled. See the Types table below. Enabled Types are a semicolon delimited list.	This list is retrieved from the configuration variable "Logging_Enabled_Types"
Include()	The information gathered for a class might be optional to actually "log" that information. Included properties are a semicolon delimited list.	This list is retrieved from the configuration variable "Logging_Includes"
LoggerEnabled()	Determines if a logger is enabled given a type. Name of enabled loggers are a semicolon delimited list. Types that are supported for a given logger are a semicolon delimited list. See the Types table below.	List of enabled loggers is retrieved from the configuration variable "Logging_Enabled_Loggers". List of types enabled for a given logger is retrieved from the configuration variable $"Logging_Enabled_{name}_Types".
HideKeys()	Listing of any custom properties that will be obfuscated in the log. Eg. Passwords. The property name will be the "key" of the item inserted into one of the dictionaries that will be read by the logger. Property keys are a semicolon delimited list.	This list is retrieved from the configuration variable "Logging_HideKeys"
MaxLength()	Maximum length of a custom property to be logged given a logger. This is useful for not logging an image, or large set of text.	1000: Also could be retrieved from the configuration variable $"Logging_{loggerName}_MaxLength"

Existing Types (You can also create your own "Type" - see Customizations):

Type	Info Class	Statement	Severity
API	ApiInformation	ApiEntry()	Information
Attribute	AttributeInformation	AttributeEntry()	Verbose
Database	DatabaseInformation	DatabaseEntry()	Verbose
Exception	ExceptionInformation	Exception()	Error
Hidden Exception	ExceptionInformation	HiddenException()	Warning
HTTP Response Error	HttpResponseInformation	None - see Alerts	Warning
Long Running	LongRunningInformation	None - see Alerts	Warning
Message	MessageInformation	Log()	User-Supplied
Middleware	MiddlewareInformation	MiddlewareEntry()	Information
Timer	TimerInformation	TimerStart()	Information
Unauthorized	UnauthorizedInformation	Unauthorized()	Warning
Web Service	WebServiceInformation	WebServiceEntry()	Verbose
Developer Supplied	CustomInformation1 CustomInformation2 CustomInformation3 CustomTimerInformation1 CustomTimerInformation2 CustomTimerInformation3 Custom Custom	Custom1() Custom2() Custom3() CustomTimer1Start() CustomTimer2Start() CustomTimer3Start() Generic() Entry()

Customizations

Besides having most features be highly configurable, you may wish to customize the logger in any number of ways.

Adding things to be logged

You can extend any of the BaseLogInformation classes to add properties. If you want to update or remove properties, you can update these in the SetProperties or UpdateProperties methods.

Once you have added a property by extending the intended class, you can register your own class instead of the default class.

Eg.

    // Extend a given class by adding a property
    public class MyNewExceptionInformation : ExceptionInformation
    {
        private string MyNewProperty { get; set; }
        public MyNewExceptionInformation(ILoggerInformation info, IApplicationSettings appSettings) : base(info, appSettings) { }

        public override void Initialize(Exception ex, string title, bool messageOnly)
        {
            base.Initialize(ex, title, messageOnly);
            MyNewProperty = "Some Value";
        }

        public override void SetProperties(ILoggerConfiguration config)
        {
            base.SetProperties(config);
            MedProperties.Add("My New Property", MyNewProperty);
        }
    }

    // Register this so that this class everywhere in place of ExceptionInformation (startup.cs)
    services.AddTransient<ExceptionInformation, MyNewExceptionInformation>();

Design-time capabilities

If you want to keep things cleaner, you can create your own logging class. To do this, you'll first inherit from one of the following "custom" classes:

All of these interface methods take a generic type as the argument with an optional message. The generic type gets saved and you will, in your class, handle the logging of this generic object by assigning all of it's properties into the proper dictionary (High/Med/Low properties).

You will then add a service registration to specify that your class will be used when calling the proper method on the ILogger interface. Note that when you wish to log something, this will use the "Custom()" methods on the interface. This method takes a generic object, which may or may not be the same type as you are expecting, so there is a potential for run-time anomolies.

If you wish to have a better name and work around these anomolies, it's recommended that you:

Create your own interface with the proper method signature - this interface should inherit from ILogger
Implement this method in a class which inherits from Logger
The implementation should simply forward on the call to the proper "Custom()" method.

Eg.

    // Create your class which will act as the generic type
    public class Widget
    {
        public int Id { get; set; }
        public string Name { get; set; }
    }

    // Create your logging class which inherits from one of the CustomInformation classes
    public class WidgetInformation : CustomInformation1<Widget>
    {
        protected WidgetInformation(ILoggerInformation info, IApplicationSettings appSettings) : base(info, appSettings) { }
        public override string Type => "Widget"; // Required override
        protected override TraceEventType CustomSeverity => TraceEventType.Verbose; // Required override
        protected override string CustomMessage => "A widget was created"; // Optional override which means this will be used instead of a user-supplied message
        // Save off any properties you wish to have recorded - or you could just serialize the entire object into one of the dictionaries
        protected override void SetHighProperties(Dictionary<string, string> highProperties, Widget obj) => highProperties.Add("Name", obj.Name); 
        protected override void SetLowProperties(Dictionary<string, string> lowProperties, Widget obj) => lowProperties.Add("Id", obj.Id.ToString());
        protected override void SetMedProperties(Dictionary<string, string> highProperties, Widget obj) { }
    }

    // Service registration (startup.cs)
    services.TryAddTransient<CustomInformation1<Widget>, WidgetInformation>();

    // Test call
    ILogger logger; // Should be injected
    await logger.Custom1(new Widget { Name="Foo", Id=1 });
    await logger.Custom1("a string object"); // This will compile, but will be a run-time exception with service lookup

    // Alternative: Inherit and extend the interface
    public interface ICustomizedLogger : ILogger
    {
        Task Widget(Widget widget);
    }

    // Implementation
    public class CustomizedLogger : Logger, ICustomizedLogger
    {
        public CustomizedLogger(ILoggerConfiguration config, ILoggers loggers, IServiceProvider provider, IAlert alert, LongRunningInformation longRunningInfo, HttpResponseInformation httpResponseInfo, IApplicationSettings app, IEmail email)
            : base(config, loggers, provider, alert, longRunningInfo, httpResponseInfo, app, email)
            { }

        public void Widget(Widget widget) => Custom1(widget);
    }

    // Registration (Note that the ILogger can still be used, but if you want to use your customized calls, you should use this one)
    services.TryAddTransient<ICustomizedLogger, CustomizedLogger>();

    // Make the new call
    ICustomizedLogger logger; // Should be injected
    logger.Widget(new Widget { Name="Foo", Id=1 });

Additional Design-time capabilities

You only create 3 Insert-Only custom classes, and 3 Updateable custom classes. If you need more, then you'll have to take a slightly different approach. Or, if you need to customize a class that will take more than 1 object as a parameter (You can always make this a single object by using a container object).

The approach is similar to the custom classes, except that you'll be directly inheritied from BaseLogInformation, or from TimerBaseInformation (for timers). This class should be (see below for example):

Contain all the parameters/objects you wish
Have all custom properties settable via a single method (eg. Initialize() which will first call InitializeBase())
Ensure the custom properties are sent to the loggers via SetProperties()
This class itself should be passed in to the proper method
1. Non-Timer: Generic()
2. Timer: Entry() and Exit()

Eg.

    // Create your class inheriting from BaseLogInformation
    public class WidgetInformation : BaseLogInformation
    {
        public override string Type => "Widget";
        public string Name { get; private set; }
        public int Id { get; private set; }
        public WidgetInformation(ILoggerInformation info, IApplicationSettings appSettings) : base(info, appSettings) { }

        // Best practice to do all initializing/setting in a method
        public virtual void Initialize(string name, int id)
        {
            InitializeBase(TraceEventType.Information);
            Message = "Something happened with a widget"; // Optional to set this
            Category = "Widgets"; // Optional to set this
            Name = name;
            Id = id;
        }

        // The following method is where you'll save your properties into a dictionary which will be logged
        // Note: There is an UpdateProperties() that will be called for timers to reset this information (base method clears the dictionaries)
        public override void SetProperties(ILoggerConfiguration config)
        {
            base.SetProperties(config);
            HighProperties.Add("Name", Name);
            LowProperties.Add("Id", Id.ToString());
        }

    // How to use this
    ILoggerInformation info; // Should be injected
    IApplicationSettings app; // Should be injected
    var widgetInfo = new WidgetInformation(info, app);
    widgetInfo.Initialize("my name", 1);

    ILogger logger = null; // Should be injected
    await logger.Generic(widgetInfo);

    // No service registration necessary, which is why you "new" up an instance