How to monitor if a meta index key is full in the NetWitness Platform

Issue

Each meta key that is indexed in the RSA NetWitness Platform has a valueMax value associated with it. This is the maximum number of unique values that can be stored in the index for this meta key.

This is defined in the index-concentrator.xml and index-concentrator-custom.xml values on a Security Analytics concentrator.

For example:

<key description="Hostname Aliases" level="IndexValues" name="alias.host" format="Text" valueMax="2500000" />

This shows that the alias.host meta key can contain up to 2500000 unique values in an index slice.

If this value is exceeded, then the additional unique values will not be able to be seen when investigating, although it will still be recorded in the session information.

It is therefore important to be alerted if the index for the meta key is full.

In some cases, the number of unique values for a key may exceed any setting of ValueMax that is used. For example, if you were to index URLs seen by the system then the index for this meta key would become quickly full due to the large number of unique possible values. For source ports and destination ports for a TCP session then there is a maximum of 65536 possible values so the valueMax is set to this value.

When a meta key is full the following will be seen in /var/log/messages on the concentrator:

Sep 18 16:40:20 logconc nw[11922]: [Index] [warning] Index key alias.host has reached max capacity of 2500000 values and will ignore new values for this slice.