The Raven Report

Ok so now we know what to look for how to we look for it?

Tools

Task Manager

Task manager can quickly provide general memory information. On the performance tab, you can quickly determine how much physical memory is in the system, how much physical memory is available, Kernel memory usage and how much memory the VMM has promised to running processes. With this information you can determine if further investigation is required.

The processes tab also has a wealth of information, but first you have to enable additional columns via the View à Select Columns option. On this tab you can quickly determine the amount of memory a process is using and how much of the paged & nonpaged pool is being used.  There many addition columns available, so don’t settle for the default view.

Process Explorer does everything task manager does and more. You can really dig into a process and figure exactly what it’s doing and what kind of resources it’s consuming.

Get it here:  http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Poolmon is a great tool for tracking down kernel memory leaks or just determining what is consuming kernel resources .

More info here: http://support.microsoft.com/kb/177415  
& here: http://technet2.microsoft.com/windowsserver/en/library/0d302498-c947-4655-95af-719ae75acfb51033.mspx?mfr=true

Performance Counters

Memory\Available Mbytes (MB):  Indicates the amount of physical memory (in MB) immediately available for allocation to a process or for system use. Used with the Pages/Sec to determine if the system is low on physical memory.

Memory\Pages/sec: Pages/sec is the number of pages read from the disk or written to the disk to resolve memory references to pages that were not in memory at the time of the reference (hard page faults), and the number of pages that were redirected to memory outside of the working set.  This counter also includes paging traffic on behalf of the system Cache to access file data for applications, so a large number alone is not reason for concern just further investigation. This is the primary counter to observe if you are concerned about excessive memory thrashing.

Memory\Pool Nonpaged Bytes: This counter shows the current size of the Nonpaged pool in Bytes. Anything over 200MB should be investigated, and unless memory tuning options are in place (/3GB) then anything over 100MB.

Memory\Pool Paged Bytes: This counter shows the current size of the paged pool in Bytes. In general anything over ~250MB should be investigated unless memory tuning options are in place (/3GB) then anything over 150MB.

Memory\Free System Page Table Entries: This counter shows the number of page table entries that are available for use. Anything under 8,000 needs a closer look.  If the system is getting low consider using the /userva=XXXX with something like 3030 to start to bump these up.  

Memory: Cache Bytes: This is the size of the system cache. You can use this counter with the Pool nonpaged/paged & PTEs to determine who kernel memory is being allocated.

Registry Keys 

As with all registry modifications, use caution. Under normal circumstances you should not need  to modify these keys, but if you are running into kernel memory issues proper use of these keys can resolve the problem.

NonPagedPoolLimit: Specifies the maximum amount of system VA space that can be used by the nonpaged pool.

PagedPoolLimit: Specifies the maximum amount of system VA space that can be used by the paged pool.

SystemCacheLimit: Specifies the maximum amount of system VA space that can be used by the system cache.

SystemPtesLimit: Specifies the maximum amount of system VA space that can be used by I/O mappings and other resources that consume system page table entries (PTEs).

SessionSpaceLimit: Specifies the maximum amount of system VA space that can be used by session space allocations

PoolUsageMaximum: Determines the Max Poll usage before pruning starts.  The default is 80%. A value of 40-60 provides a workable solution to some paged pool exhaustion issues.

Windows 2003 64Bit can support a theoretical limitation of 16 exabytes or 2⁶⁴

·         Kernel Mode Address space - 8 TB

·         Paged Pool – 128GB

·         NonPaged Pool – 75% of RAM up to a maximum of 128 GB.

·         System Cache/PTEs - 1 TB regardless of physical memory.

As you can see, with 64bit Windows all of the previous memory constraints are blow out of the water.   For just this reason alone upgrading to a 64bit OS should be seriously considered.

Troubleshooting 32bit memory problems

For those who have legacy applications and 32bit hardware upgrading may not be an option. So in the following I will go over some tips and tools to identify memory bottlenecks a 32bit system.

Physical Memory Exhaustion - This is pretty much the straight forward if you are out of physical memory and the system is excessively paging to disk. More physical memory is most likely the answer.

Kernel Memory Exhaustion -  Memory problems in  kernel memory are a little more challenging to nail down.  Here are some of the signs:

Paged Pool & Nonpaged Pool

·         Sluggish or unresponsive user interface

·         Server has message or client processing failures

o   RPC Service Unavailable

o   Not enough Storage to process this command

·         Pool allocation failures:

System PTEs

·         Server fails to respond to I/O requests

·         Server fails to respond to network requests

·         Server has message or client processing failures

If you are experiencing these problems or other unexplained issues kernel memory could be your issue.

Your first stop should be the performance tab in Task Manager. From here you can quickly gauge if you are approaching the kernel memory limits of the system. Take a look at the Kernel Memory section and look at the amounts used by the page pool ( > 250MB or > 150 with /3GB ) and nonpaged pool( >200 or >100 with /3GB).  If we come up higher than these numbers further investigation is required.   We’ll start there next time.