Goldengate Memory Utilization - Script

Automated Script to get Goldengate process Memory Utilization. 

Below script is a Plug and Play tool for getting memory Utilization of Goldengate Process running on HP-UX or Linux Machine. 

For Creating Script in your Environment do vi GG_MemoUtilization.ksh copy and pest below code


#!/bin/ksh

###########################################################
# +----------------------------------------------------------------------------+
# | Technology : Oracle Goldengate                                                 |
# | Orther      : Kamlesh Parmar (Kamleshparmar21@gmail.com)       |
# | FILE          : GG_MemoUtilization.ksh                                         |
# | PURPOSE   : This Script is Useful to get the Memory Utilization of   |
# |               Goldengate Extract and Replicat Process                     |
# | PARAMETERS   : None                                                              |
# | EXAMPLE      : sh GG_MemoUtilization.ksh                                   |
# +----------------------------------------------------------------------------+
###########################################################

###############################
# determine the OS type
###############################
OSNAME=`uname`

case "$OSNAME" in
  "HP-UX")
    echo "OSNAME = $OSNAME"
    ;;
  "Linux")
    echo "OSNAME = $OSNAME"
    ;;
  "*")
    echo "This script has not been verified on $OSNAME"
    exit 1
    ;;
esac

###############################
# set the temp file
###############################
TMPFILE=/tmp/ggmem.tmp
if [ -f $TMPFILE ]
then
  rm -f $TMPFILE
fi

################################
# loop over the gg process types
################################
PROCESSES="extract replicat"

for PROCESS in $PROCESSES
do
FLAG=""
FLAG=`ps -ef | grep $PROCESS | grep -v grep`

if [ -z "$FLAG" ]
  then
    echo
    echo
    echo
    echo "#####################################"
    echo "#---No $PROCESS processes found-----#"
    echo "#####################################"
    echo
    echo
    echo
 else
    echo
    echo
    echo "###########################################################"
    echo "#-------Individual $PROCESS Process Memory Usage--------- #"
    echo "###########i###############################################"
    echo
    case "$OSNAME" in
      "HP-UX")
        UNIX95=1 ps -e -o user,pid,vsz,sz,etime,args | grep -w $PROCESS |grep -v grep > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{ printf "%3.4f %s\n" , $3/1024/1024,"GB   "$8}' | sort -r
        ;;
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{print $2/1024/1024, "GB", $12}' | sort -k 2
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac


    echo
    echo
    echo
    echo "#####################################"
    echo "#---Total $PROCESS Process Usage----#"
    echo "#####################################"
    echo
    case "$OSNAME" in
      "HP-UX")
        cat $TMPFILE | grep $PROCESS | awk '{count ++; sum=sum+$3; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/1024/count, "GB"; \
          print "Total memory usage       =", sum/1024/1024,   "GB"}'
        ;;
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{count ++; sum=sum+$2; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/1024/count, "GB"; \
          print "Total memory usage       =", sum/1024/1024,  " GB"}'
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac
    rm -f $TMPFILE
  fi
done
echo
echo
exit 0 

Memory Utilization of Goldengate Process

Why GG process consumes memory ?

The Oracle redo log files contain both committed as well as uncommitted changes but GoldenGate only replicates committed transactions. So it needs some kind of cache where it can store the operation of each transaction until it receives a commit or rollback for that transaction. This is particularly significant for both large as well as long-running transactions.

This cache is a virtual memory pool or global cache for all the extract and replicate processes and sub-pools are allocated for each Extract log reader thread or Replicate trail reader thread as well as dedicated sub-pools for holding large data like BLOBs.

Documentation states: “While the actual amount of physical memory that is used by any Oracle GoldenGate process is controlled by the operating system, the cache manager keeps an Oracle GoldenGate process working within the soft limit of its global cache size, only allocating virtual memory on demand.”

Parameters to control memory usage in GG

CACHEMGR CACHESIZE {size}

CACHEMGR CACHEDIRECTORY {path} {size}

CACHEMGR CACHEBUFFERSIZE {size}

The CACHEMGR parameter controls the amount of virtual memory and temporary disk space that is available for caching uncommitted transaction data.

The CACHEMGR CACHESIZE parameter controls the virtual memory allocations and in GoldenGate versions 11.2 onwards for a 64-bit system the CACHESIZE by default is 64 GB.

While the CACHESIZE parameter controls the Virtual Memory, if that is exceeded then GoldenGate will swap data to disk temporarily and that is by default being allocated in the dirtmp sub-directory of the Oracle GoldenGate installation directory.

The dirtmp location will contain the .cm files. The cache manager assumes that all of the free space on the file system is available and will use it to create the .cm files until it becomes full. To regulate this we can use the CACHEMGR CACHEDIRECTORY parameter and provide both a size as well assign a directory location where these .cm files will be created.

The sizes of the initial and incremental buffers are controlled by the CACHEBUFFERSIZE option of CACHEMGR.

To know the Memory Utilization of Goldengate Process in your Environment use below Automated Script

Automated Script to know Memory Utilization of Goldengate Process