emeadaxsupport: Possibilities to create Memory Dumps from crashing processes

[Blog bot]

Источник: http://blogs.msdn.com/emeadaxsupport...processes.aspx
==============

Sometimes it happens that during regular operation the Dynamics AX Application Object Server (AOS) is suffering from an abnormal program termination, which is better know as a crash. Users connected with the Dynamics AX Client to the affected AOS Instance are noticing such an issue very soon as they get disconnected from the AOS.

In such a situation there is usually not much you can do, maybe trying out a more recent Dynamics AX Kernel Hotfix or Roll-Up.

When you decide to open a Support Request to get help identifying what concretely causes the AOS to crash, you will get asked to create a Memory Dump of the crash, and if you have a Memory Dump ready by that this will speed up the process a bit.
 

What is a crash and why do I need to collect a Memory Dump?

At a very top level a process is crashing when it is performing an illegal operation that is not captured / handled by the process itself. This unhandled exception forces the Operating System to terminated the process. Examples for an illegal operation are for example a division by zero or accessing an invalid memory location.

When you are creating a Memory Dump from a crashing process (Crash Dump) at a high level the dump is a snapshot from the process’ memory and register context (together with some more information) at the time of the crash. So the dump shows for example the very last operation being executed that caused the process to crash.

By doing a post mortem analysis of the memory dump it is (usually) possible to identify what the AOS was doing (e. g. was X++ code executed) when the crash happened, so the memory dump is the key to solve a crash situation! 
 

Did the AOS really crash?

Before I go into detail about how to create a Memory Dump I would like to show you however some possibilities to confirm you are definitely running into a crash situation and here the Windows Event Log of the AOS is the key.
 


Unexpected termination

The most common event you see in a crash situation is the Error event 1000 from Application Error in the Application Event Log. This is definitely a confirmation that an AOS Instance was terminating unexpectedly.

Event Type: Error
Event Source: Application Error
Event ID: 1000
Description:
Faulting application Ax32serv.exe, version 4.0.2163.0, time stamp 0x48b574e5, faulting module Ax32serv.exe, version 4.0.2163.0, time stamp 0x48b574e5, exception code 0xc0000005, fault offset 0x001630bf, process id 0x1a90, application start time 0x01c944075355b99b.
Sometimes there is no entry in the Application Event Log however only in the System Event Log. The Error event 7034 from Service Control Manager is another confirmation that an AOS Instance was terminating unexpectedly.

Event Type: Error
Event Source: Service Control Manager
Event ID: 7034
Description:
The Dynamics Server$01-AOS1 service terminated unexpectedly. It has done this 2 time(s).
 

Forced termination

In contrast to an unexpected termination there is also a forced termination whose effects are however similar not to say the same. The difference to an unexpected termination is that a forced termination is caused by the program logic of the Dynamics AX AOS Kernel itself because some error checking code determined a situation the AOS cannot resolve; however continuation in such a situation would most likely cause some more serious effects up to data corruption, so it is better to terminate at this stage and log this fact to the Application Event Log.

In such a case an Error event 110 from Dynamics Server is logged having the wording “Unexpected situation” in the Description.

Event Type: Error
Event Source: Dynamics Server 01
Event ID: 110
Description:
Object Server 01: Unexpected situation
More Information: Session Allocation Failed: Session is already allocated.
 

Controlled shutdown

We have seen so far confirmations for an unexpected termination and a forced termination, however there exist also proves for a controlled shutdown where the AOS in fact did not crash.

The Information event 108 in the Application Event Log is always logged when the AOS has been requested to stop – in a controlled way. If you see this event someone or something has requested the AOS to stop by intention.

Event Type: Information
Event Source: Dynamics Server 01
Event ID: 108
Description:
Object Server 01: Dynamics Server has been stopped
The Information events 7035 and 7036 from Service Control Manager in the System Event Log are another prove of a controlled shutdown of the AOS service. Again someone or something requested to AOS to stop by intention.

Event Type: Information
Event Source: Service Control Manager
Event ID: 7035
Description:
The Dynamics Server$01-AOS1 service was successfully sent a stop control.


Event Type: Information
Event Source: Service Control Manager
Event ID: 7036
Description:
The Dynamics Server$01-AOS1 service entered the stopped state.  
 


What do I need to create a Crash Dump?

A Crash Dump is a Memory Dump that is automatically created when a process is executing an operation (CPU instruction) that causes the process to crash.

Lets assume you have confirmed your AOS is indeed crashing and you need to create a Memory Dump (Crash Dump).

There exist several tools you can use to create a Memory Dump from a crashing process, and even the operating system is providing some built in functionality.
 


ADPlus

To create a dump with ADPlus you first need to download and install the Debugging Tools for Windows and launch adplus.vbs from the command line with the -crash and -pn ax32serv parameters.

adplus.vbs –crash –pn ax32serv
After executing the script you will see new command windows showing up – one command window per running AOS Instance. Actually this are no command windows but this is the command line debugger CDB that is attached to the AOS Instance.

It is very important that you don’t close those CDB instances nor log off from the machine because a closing of the CDB instances will also terminate the attached AOS Instance!

Please note:
ADPlus will only attach to the currently running AOS Instances. If you start another AOS Instance afterwards or restart a running AOS Instance you need to execute ADPlus again!

A more detailed instruction about ADPlus can be found in the KB article How to use ADPlus to troubleshoot "hangs" and "crashes".
 


Debug Diagnostic Tool v1.1 (DebugDiag)

To create a dump with the Debug Diagnostic Tool v1.1 (DebugDiag) you need to download and install it first.

When you launch the tool you are presented with a Graphical User Interface (GUI) and a Wizard the helps you to create a new Crash Rule.

The advantage of using DebugDiag over ADPlus is that DebugDiag has a GUI and that DebugDiag with the Crash Rule created not only attaches to the currently running AOS Instances but also AOS Instances started after the creation of the Crash Rule.

Please note:
DebugDiag is not supported running on Windows Server 2008 and Windows Server 2008 R2!

A more detailed instruction about the Debug Diagnostic Tool v1.1 can be found in the following article How to Use the Debug Diagnostic Tool v1.1 (DebugDiag) to Debug User Mode Processes.
 


Windows Error Reporting (WER)

The Windows Error Reporting (WER) that is built-in in Windows can be (in Windows Server 2008 and higher) configured to automatically create and store memory dump from an AOS crash.

The configuration happens by making some Registry modifications, more concrete creating the following Registry Key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\Ax32Serv.exe
In this new Registry Key several Registry Values have to be added as well (DumpFolder, DumpCount, DumpType, CustomDumpFlags).

Please note:
Do not set the DumpType to 1 or 2, but set the DumpType to 0 and set CustomDumpFlags to 137254 (decimal).

A detailed instruction about the needed Registry Values is described in the following article Collecting User-Mode Dumps.
 

Crash or High Memory Usage?

As indicated before a (Full) Memory Dump is containing the process’ memory and register context, so usually the file size of the dump file is about the size of the currently allocated virtual memory.

If the file size of the memory dump is about or larger than 1.6 GB and your are not running a 64 bit version of Dynamics AX the chances are very high the crash was caused by a High Memory Usage situation or because of a Memory Leak.

In this case please also have a look at my earlier post Why does the server need to free resources and terminates client sessions?



Источник: http://blogs.msdn.com/emeadaxsupport...processes.aspx