Being Forensic

Since our inception in 2003, I have been asked many times what is behind our name–Forensic.  What does it mean with respect to IT?

To be honest, it is always a difficult question to answer in just a few words.  I start talking about troubleshooting computers or computer systems and as soon as I bring up computers, I usually get a comment like:

“…oh, I get it now, my nephew is in computers too.  He set up my Wi-Fi network…”.

They got their answer–we just work in computers.  Smh.

Sometimes I just drop it and other times I continue to try and explain how we are different.  Yes, we work in computers, but that has nothing to do with being Forensic, or really what we do.  Computers just happens to be the current platform we work in.

So, since I have you captive for a few more words anyway, let me explain.  Let’s start with a web search on the definition of “forensic”.

The word “Forensic” according to the definition above (remove the crime part for two reasons, 1-we don’t currently work in the criminal justice system, and 2-we are not criminals) means “…the application of scientific methods and techniques in the investigation…”.

Our company works with “IT” which is short for “Information Technology”.

Replace crime with IT and you have:

“Forensic IT – The application of scientific methods and techniques in the investigation of Information Technology”

Now we are getting somewhere.

I won’t bore you with more web searches but I will just tell you that the scientific method means

“…the systematic observation, measurement, and experimentation, and the formulation, testing, and modification of a hypotheses…”. 

Oddly enough, we can apply that to almost anything.  Any problem.  Any industry–yes, even the criminal justice system.

A simple example.  If you wake up and cannot find your car keys, you start to apply the scientific method.  You do this naturally without thinking very hard.  You think back to when you last had them and piece together what you did between when you last had them and the present.  You think of what activities you did, people you talked with, places in your house you went, and usually, in a short time, you find your keys.

You don’t do silly things like look in the attic if you did not go in the attic.  If you did not go out on your deck or your back porch then you probably don’t go looking there either.  You use systematic observation to help find your keys.

We are no different at Forensic IT.  We pride ourselves on looking at problems in this same fashion.  It starts with learning some basic architecture of how a process works, how an application communicates, and what the anatomy of it is.

If we think of the above simplified process and break it down generically we can say that most computer processes do the same four common things:

  1. Use memory to hold data
  2. Interact with a user, system, or other process or device.
  3. Follow a routine or algorithm for a purpose–i.e., the program.
  4. Read/write data to files, networks, other processes or devices.

Now, that is a generic view of a process, but you have to admit–in its simplicity, it covers all processes.  Not all processes use all four items listed and a few might add more, but in general it covers is.

Since all processes do basically the same core things, we felt that we could add value to all customers by tailoring our skills to analyze, expose, troubleshoot, and explain how processes do those four items.

I was teaching our Advanced Windbg course to a 3rd level support group and we were focusing on how the Virtual Memory Manager in windows mapped and addressed memory.  I was challenged by one of the attendees who said:

“…so you mean to tell me that you don’t know all the differences between windows 2003, 2008, and 2012 heap managers?  I thought you would be an expert on that…”

Challenge accepted!

My response was to attach a debugger to a windows 2008 server that they had and use Windbg to map out the heap manager, put breakpoints on heapalloc() and heapfree() APIs and proceed to draw a heap manager architecture on the board in more detail than he could google about it.  I hope that I handled myself in a professional way, but I do remember feeling a bit too satisfied during the challenge!

I bring this up to serve a point on being forensic.  We do not pride ourselves on knowledge on how specific things work, rather, we pride ourselves on being able to analyze, reverse engineer, explain, and otherwise expose the architecture on how something works.  To us, that is a way more valuable skill than trying to read and memorize technical details about any operating system or process.

Does this mean that we don’t believe in becoming experts on how certain processes or systems work?

Not at all.  We strongly encourage our engineers to attain as much knowledge of how things work as possible.  The difference is that we want them first to build skills to attain the knowledge–not the knowledge itself.

Below are two simple pictures that hopefully explain in more detail what a non-forensic subject matter expert (a level-2 or level-3 support engineer) sees in a process or system versus a forensic one.  Both have value and both have a place.

If you want quick answers on how to configure or troubleshoot connectivity or look at normal application failures, the application or system expert can solve most of these issues.  They have the knowledge, experience, and understanding of the system to help.

See below.

Figure: Level 2 or Level 3 Application Support

In this picture, we see a typical Level 2 or Level-3 support engineer with lots of system experience.

They are great at connectivity, configuration, and normal system failures.

They definitely provide value to an organization and fulfill a vital role in system support.

The troubleshoot many process and system failures.

When all processes are running but the system doesn’t work or when nothing is written to the log files or an issue surfaces that the specific application knowledge, experience, and internal architecture does not solve, the Forensic Engineer can help.  They look at the same picture but with a different view, a different perspective.

Not better, just different.

Figure:  Forensic Engineer

The Forensic engineer looks at the same picture using both a telescope and a microscope at the same time.

Although they have a generic view of what the application is intended to do (i.e, it’s requirements or design specification), they have a detailed view of what the application is actually doing.

Not only do they see the minute details of a process and system down to the memory blocks and disassembled code, they see a macro view of the overall system.   It is with this different view, and frankly, without internal system knowledge of the process itself that they excel and provide value.  This is because they can look at a failure and examine data to explain how the process itself behaves and operates.

Without any preconceived notion or expectation of behavior, they are free to let the data (The Bits & Bytes) lead them down a path to root cause.

I would be remiss if I did not point out that when you use a forensic, scientifically backed method or approach, you can safely declare:

A Forensic Engineer:

  • Never guesses.  They explain behavior with data.
  • Always has a logical next step in the troubleshooting process.
  • Root cause is certain.