Optimization

Web Protection Performance Tuning

Web Protection can be deployed on customer applications that may have different throughput and scale requirements. Various configuration knobs are provided to cater to these requirements.

The provided configuration knobs are:

1. vIPC Queue Size
2. Number of vRule Worker Thread

vIPC Queue Size Configuration

The vIPC queue is a shared memory queue that is used to exchange data plane messages. A queue is created as per application process requirements. In case the configured application creates many ephemeral processes, then it is optimal to reduce the vIPC data queue size.

1. The default data queue size is set to 26 MB in one direction. This means that 10 application processes would consume 520 MB of RAM (including both directions)
2. However, if the application creates hundreds of ephemeral processes, then the memory allocated for vIPC queues will exceed 520 MB. In these scenarios, it is optimal to reduce the value of the parameter dataQueueSizeto ensure a reasonable memory consumption. For example, if there are 100 application processes, this value must be modified to 5MB to keep the total memory consumed under 500MB. 
   1. In cases of Apache/Ngnix servers where processes are spun up in large numbers, the default data queue size of 26 MB can be reduced to a lower value (Example: 5 MB). This configuration can be applied to all processes. Configuration of this parameter is required for PHP processes
   2. In cases where the maximum number of processes spun up by a web server is configured to be 100 or more, then the default data queue size of 26 MB can be reduced to a lower value like 5 MB
3. Configure the vIPC queue size using the parameter dataQueueSize using the below command:

   vsp-cli config vIPC-server edit dataQueueSize <desired_size_in_bytes>

4. The command top provides the memory consumed by each process. If the resident memory of the vIPC-server goes beyond 512 MB during a load condition, then it is an indication that vIPC queue needs resizing. It is suggested to reduce it by 50% for each iteration.  This procedure must be repeated as many times as necessary to limit the vIPC memory consumption to a reasonable amount
5. Typically, vIPC queue size adjustment is done along with vRule worker thread adjustment 

vRule Engine Worker Thread Configuration

1. vRule Engine deploys a number of worker threads to handle Application data traffic. The default value is typically configured to two threads. Each worker thread requires 1 vCPU
2. However, if an Application requires more throughput, then it is recommended to increase the number of worker threads in vRule. 
3. The number of worker threads and memory can be configured by executing the commands below:

   vsp-cli config ae edit numWorker <1-16> --persist  #To modify number of threads
   vsp-cli config ae edit autoscaling false #To disable autoscaling

4. When the application throughput requirement is more than the supported value, it creates a congestion in the vRule Engine and as a result, packets are dropped by vRule’s flow controls. The number of packets dropped are logged in the vRule stats file
5. The vRule stats are available in the file: /vspstats/vrule/vrule_stats. The below diagram depicts a part of the vRule stats file
6. If the drop count increases continuously, then it is a clear indication that more worker threads are needed. There should be a continuous increase in the drop count
7. Received M indicates the accumulated number of data plane messages. The rate of increase of these messages indicates the overall throughput rate at which the vRule Engine is processing data packets
8. The average packet size and throughput can also be obtained from CMS dashboard

VSP Restart

A change in the vIPC queue size or the vRule Engine worker threads requires a restart

1. Restart the vRule Engine using the below command:

   vsp-cli restart ae 

2. Restart the vIPC server using the below command. This command restarts all the services as the vIPC-server cannot be restarted in isolation

   vsp-cli restart vIPC-server

Memory Optimization for httpd Server

When the httpd server is instrumented by VSP Memory, decrease the VSP Memory overhead by configuring the below settings in the file httpd.conf:

1. MinSpareServers 256 
2. MaxSpareServers 256 
3. MaxRequestsPerChild 0

 In some configurations, httpd spawns multiple short-lived worker processes that handle only a single httpd request. VSP-Memory is non-optimal for short-lived processes. By applying these settings, httpd spawns a pool of worker processes that stay alive for the lifetime of httpd. This allows httpd under VSP-Memory to maintain near-native throughput.

Memory Compatibility for RTM (Linux)

In Linux systems, if the RTM feature is enabled, VSP-Memory may not function as expected. This is defined by the environment variable glibc.elision.enable. This is a global setting at the Operating System level. The setting is dependent on the below factors:

1. Operating System and version
2. glibc library version

Below table provides the VSP Memory Compatibility for RTM

CMS Configuration Limits

[Version 3.0.0 and Above]

This section provides the CMS configuration limits. The values provided below may not be hard limits in many cases, but exceeding them may lead to an unstable CMS in the deployed environment. Virsec highly recommends staying within these limits. Reach out to the Virsec technical team in case your setup/environment requires scaling beyond these numbers.

VSP Probe - Memory and CPU Monitoring (VMs only)

VSP-Manager monitors its resource usage to ensure that the VSP processes do not overuse a customer machine’s resources. The CPU and Memory Usage are monitored on both Linux and Windows VMs. If the maximum limit is crossed, the Probe is stopped.

1. The resource utilization is monitored after the configured time interval elapses
2. Modify the time frequency (Usage Sampling Time) between each check using the below command:

   vsp-cli config vsp-manager edit usageCheckRate <time_in_seconds> --persist

3. If the utilization exceeds for a configured threshold, the utilization is monitored again after the time frequency. The maximum number of times the utilization is allowed to exceed the threshold (Usage Sampling Rate) can be configured:

   vsp-cli config vsp-manager edit usageThreshold <Maximum_Allowed_Attempts> --persist

4. Once this count is reached, the VSP Probe is stopped and its status is Disconnected in CMS
5. The configurations can be set dynamically at runtime
6. If the values are set to zero (0), the monitoring is turned off
7. The default values are:
   1. Time frequency – 60 seconds
   2. Consecutive attempts with utilization exceeding - 3
   3. By default, the values are NOT configured at VSP process level

General Configurations

Shutdown VSP Probe or a Single VSP Service

1. VSP-Manager can be configured to shut down all the VSP Probe services OR just the problematic service. Configure the preference using the below command:

   vsp-cli config vsp-manager edit singleServiceShutdown <true/false> --persist

   1. true – To shut down only the problematic VSP Service
   2. false – To shut down all VSP services

Auto-restart a VSP Service

1. If the setting singleServiceShutdown  is true, then an auto-restart for services that are killed by VSP-Manager can be configured using the below command:

   vsp-cli edit-service <Service_Name> edit autorestartTime <Time_In_Seconds>

   1. Time_In_Seconds – Defines the wait time after which the VSP Service must be restarted
2. The available services are: AE, AE-PROXY, vIPC-server, FSM, FSM-AGENT, HMM, RMP, VSP-manager, VSP-memory-assist, VSP-web-assist and VSP-APG

Ignore Initial Usage of VSP Service

1. VSP-Manager supports a configuration that ignores resource usage for the first N seconds of a VSP service’s life. This is in cases the usage during this time might be higher than the usage during normal functioning of the service

   vsp-cli edit-service <Service_Name> edit ignoreUsageTime <Time_In_Seconds>

2. The available services are: AE, AE-PROXY, vIPC-server, FSM, FSM-AGENT, HMM, RMP, VSP-manager, VSP-memory-assist, VSP-web-assist and VSP-APG

CPU Monitoring

Total VSP Services CPU Capping

1. The default limit is 40%
2. The overall Probe limits are provided in the table below:

   Usage Type	Minimum	Maximum
   Total CPU	5	100
   Memory MB Limit	200MB	<No limit>
   Memory Percent Limit	>1	100

3. The Probe limits per component are provided in the table below:

   Usage Type	Minimum	Maximum
   Total CPU	5	100
   Memory MB Limit	50MB	<No limit>
   Memory Percent Limit	>1	100

4. Configure the total CPU usage of the VSP services using the below command:

   vsp-cli config vsp-manager edit cpuLimit <Limit_In_Percentage> --persist

   
   

Individual VSP Service CPU Capping

1. Execute the below command to set the CPU limit of an individual VSP service by executing:

   vsp-cli edit-service <Service_Name> edit cpuLimit <Limit_In_Pertecntage>

2. Configure the Usage Sampling Time using the below command:

   vsp-cli config vsp-manager edit usageSamplingTime <time_in_seconds> --persist

3. Configure Usage Sampling Rate using the below command:

   vsp-cli config vsp-manager edit usageSamplingRate <Usage_Sampling_Rate> --persist

4. The available services are: AE, AE-PROXY, vIPC-server, FSM, FSM-AGENT, HMM, RMP, VSP-manager, VSP-memory-assist, VSP-web-assist and VSP-APG
5. Example:
   1. Consider the values: usageSamplingTime = 2.5 and usageSamplingRate = 2
   2. For each CPU usage reading, VSP-Manager collects 2 samples, 2.5 seconds apart. Their average value is then calculated to procure the most accurate value of the current CPU usage
   3. The average value is divided by the number of CPU cores on the system to get the overall CPU usage of a specific process

Memory Monitoring

Total VSP Service Memory Capping

1. The default limit is 40%
2. Configure the total CPU capping of the VSP services using the below command:

   vsp-cli config vsp-manager edit memoryMBLimit <Limit_in_MB> --persist
   vsp-cli config vsp-manager edit memoryPercentLimit <Limit_In_Percentage> --persist
   

   NOTE

   AE and AE-Proxy services are excluded from the total Memory capping. Utilize the Individual Service Memory capping feature if required
   
   

Individual VSP Service Memory Capping

1. Configure the memory limits of an individual VSP service by executing:

   vsp-cli edit-service <Service_Name> edit memoryMBLimit <Limit_in_MB>

   OR

   vsp-cli edit-service <Service_Name> edit memoryPercentLimit <Limit_In_Percentage>

2. The available services are: AE, AE-PROXY, vIPC-server, FSM, FSM-AGENT, HMM, RMP, VSP-manager, VSP-memory-assist, VSP-web-assist and VSP-APG
3. VSP-Manager supports the capping of memory usage by the number of MBs and the percentage of total system usage

VSP-CLI Password (Windows)

1. Password can be set during Probe Installation
   1. Starting VSP Service using services.msc is allowed but stopping is not allowed
   2. Starting/stopping VSP services using sc command is not allowed
2. VSP-CLI has three options for password management. The options are
   1. check-password: returns true when the password is configured and false when the password is not configured for VSP-CLI during Probe Installation
   2. reset-password: To reset the password using the current password OR the reset token (generated during Probe Installation)
   3. set-password: To set the password (if not set already)

      vsp-cli -h #To view help menu

3. Password is required to modify or stop services, but not required to start any VSP service
4. When the below options are used with VSP-CLI, the user is prompted for the password (if it is already set)

   vsp-cli edit-service #(edit)
   vsp-cli update-service
   vsp-cli config #(edit, add and restore) 
   vsp-cli reinit-cms