vRule Engine Troubleshooting

Log Files

vRule Engine maintains four log files:

1. Vrule_log.yyyy-mm-dd.n.log  - For vRule state logs
2. Vrule_incident.yyyy-mm-dd.n.log – To capture all incidents generated by vRule in debug mode
3. Vrule_iaemsg.yyyy-mm-dd.n.log – To capture all message logs received from probe
4. Vrule_vcr.yyyy-mm-dd.n.log – To capture VCR message debug logs

All the logs are set to info mode. Debug logs are available only when logging is enabled in debug mode. Utilize the commands below to configure the debug mode:

vsp-cli config ae edit logLevel debug #vRule state logs
vsp-cli config ae edit logReport debug  #vRule Incident logs
vsp-cli config ae edit logMessage debug #IAE message log
vsp-cli config ae edit logVcr debug #VCR message log

vRule Counters and Statistics

vRule maintains several counters and statistics that get updated based on the configured rate (Default: 5 seconds) 

1. vRule Engine provides sufficient statistics for all messages it receives:

   vsp-cli stats ae reset 
   vsp-cli stats ae view
   

2. Worker Thread Statistics – it maintains counters for every worker thread, regarding number of incidents generated at each level (attack/threat/log/benign and exception) for every feature

   virsec_Wrk1 sqlfeature -->     Attacks:         0      Threats:         0        Logs:         0       Benign:         0   exception:         0
   virsec_Wrk1 xssfeature -->     Attacks:         0      Threats:         0        Logs:         0       Benign:         0   exception:         0

3.  All the counter types are arranged in rows, Each counter type may have more than one counter arranged under a different column.
4. Example 1: virsec_Wrk1 sqlfeature - indicates SQLi incidents generated by worker thread 1 for namespace virsec that implements CVE namespace rules
5. Example 2: custom_block_Wrk0 sqlfeature - indicates counters for incidents generated by worker thread 0, for namespace custom_blockthat implements custom deny rules
   custom_block_Wrk0 sqlfeat -->     Attacks:         0       Threats:         0        Logs:         0        Benign:         0   exception:         0                                                                                                                           custom_block_Wrk0 xssfeat -->      Attacks:         0      Threats:         0        Logs:         0        Benign:         0   exception:         0  
6. Receiver counters - maintained by data packets receiver (either SHM-based or RSocket-based for remote vRule)
7. RcvdMsg - number of messages received by vRule. The messages consist of all data packets such as HTTP request/response, appalive
   1. Schd Msg - number of messages scheduled for processing by vRule worker threads
   2. Drop - number of packets dropped due to congestion. An increase in this counter over a period of time indicates that the number of configured worker threads is not enough to cater to the peak traffic
   3. Blocked – number of times a worker thread was in congestion. If this is non-zero and keeps increasing, it indicates that the peak data rate is high and number of worker threads must be increased. If this value keeps increasing, examine the flow control counters mentioned in point 10 below
   4. Unblocked - number of times a worker thread recovered from congestion
   5. Rate (MB/s) - live rate at which vRule Engine receives data traffic
   6. AvgSize - average size of received packets
   7. Pending - number of pending messages that the worker thread needs to process
8. Message counters - Maintain a count of each message type along with min/max and avg size, as depicted below
9. The message reception depends on the type of vulnerability enabled in the system. At minimum, IAE_MSG_HTTP_REQ and IAE_MSG_HTTP_RESPONSEcounters gets incremented
   1. If CMDi is enabled, then IAE_MSG_CMD gets incremented
   2. If SQLi is enabled, then IAE_MSG_SQL gets incremented
10. Flow control counters
    1. Autoscaling - indicates whether autoscaling is enabled or not. “0” – disabled, “1” – enabled. The number of worker thread are automatically scaled up to required number. This requires vRule process restart. If autoscaling is not enabled, enable it using the command:

       vsp-cli config ae edit autoscaling true

    2. ReqdWorker - recommends the number of worker thread needed by vRule Engine. This is relevant only if vRule has moved into congestion mode (Blocked in receive counter > 0). A non-zero value indicates that during peak traffic, vRule had gone into congestion (drops > 0). ReqdWorker is the measured number of worker threads needed to handle peak traffic

       vsp-cli config ae edit numWorker <number>

    3. Rate(MB/s) - peak data rate measured during congestion

VSP-cli Commands for vRule

VSP-cli commands that can be used for vRule Engine are:

1. View the statistics: vRule statistics are located in the file: /vspstats/vrule/vrule.statsthat can be watched for live updates of these counters

   vsp-cli stats ae view

2. Reset the statistics

   vsp-cli stats ae reset

3. vRule Engine has multiple worker threads running to process the messages in parallel. The commands related to worker threads:
   1. Modify the number of worker threads. vRule Engine must be restarted after modification

      vsp-cli config ae edit numWorker <number>

   2. Autoscaling can be configured as true or false. If set to true, vRule Engine automatically scales up the number of worker threads

      vsp-cli config ae edit autoscaling <true/false>

vRule does not start

Symptom: The command “vsp-cli list” lists AE process after application provisioning is successful. If vRule is not listed, AE/vRule has not started successfully

Possible Causes:

1. JDK may not be installed in the system and vRule may not start
2. The customer platform may not have write permissions to the tmp directory needed by java

Debugging Steps:

1. Check and verify if JDK 17 is installed. Install JDK it if not already installed. Internet connection is required for JDK installation
2. Execute the command on the Linux platform to check the JDK path:

   echo $VISREC_JAVA_HOME

   1. If the path is not set to the correct directory, add below line in the file /etc/environment

      export VIRSEC_JAVA_HOME=<JDK_PATH>

   2. Example: export VIRSEC_JAVA_HOME=/usr/lib/jvm/openjdk-11-manual-installation/
3. On Windows platform, check if the environment variable VISREC_JAVA_HOME is defined 
4. On Linux, vRule can be manually started using the commands:

   cd /opt/virsec/script
   ./vrule_run.sh
   
   

   1. Verify that there are no errors in the newly created log files in the directory:  /var/virsec/log/vrule/ 
5. If vrule is still not starting and vRule log file contains the error “Could not initialize class com.sun.jna.Native”:
   1. Linux: Modify the file /opt/virsec/scripts/vrule_run.sh to add the below option (if it is not already present)

      $VIRSEC_JAVA_HOME/bin/java -server -XX:+UseG1GC -XX:InitialHeapSize=$VRE_MEM -XX:MaxHeapSize=$VRE_MEM -XX:MaxGCPauseMillis=300 -DVRE_MEM=$VRE_MEM -Djava.io.tmpdir=/opt/virsec/tmp -DNUM_VRULE_WORKER=$NUM_VRULE_WORKER -DSTATS_PATH=/vspstats/vrule/ -Djava.library.path=/opt/virsec/lib/ --add-opens java.base/jdk.internal.misc=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -DDB_CACHE_PATH=/opt/virsec/data/vrule/compiled_db/ -DVRULE_CONFIG_PATH=/opt/virsec/data/vrule/ -DVRULE_JSON_CONFIG_PATH=/opt/virsec/config/ -Dlogback.configurationFile=/opt/virsec/data/vrule/logback.xml -DVIRSEC_VSP_SKU_TYPE=$VIRSEC_VSP_SKU_TYPE -jar  /opt/virsec/bin/vrules_engine-1.0.jar

   2. Windows: Modify the file: C:\Program Files (x86)\virsec\vrule\scripts\vrule_run.bat to add the below option (if it is not already present)

      set VO8=-Djava.io.tmpdir="%VSP_HOME%\tmp"
      "%VIRSEC_JAVA_HOME%\bin\java" -server -XX:+UseG1GC -XX:InitialHeapSize=%VRE_MEM% -XX:MaxHeapSize=%VRE_MEM% -XX:MaxGCPauseMillis=300 -DVRE_MEM=%VRE_MEM% -DNUM_VRULE_WORKER=%NUM_VRULE_WORKER%  --add-opens java.base/jdk.internal.misc=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true %VO8% %VO7% %VO3% %VO1% %VO2% %VO4% %VO5% %V06% -jar  "%VSP_HOME%\vrule\bin\vrules_engine-1.0.jar"
      

6. Check if Host Monitoring is enabled in Protect mode and if any HMM incident is generated with java (vRule process) getting killed. In such cases, add the appropriate exclusion rules in CMS
7. Run the command below to check if vRule process is running:

   ps aux | grep vrules

   
   

Web Application does not move to Normal

If vRule process is running but the application does not move to Normal, follow the steps in this section to debug the scenario:

1. Check if appalive counters are getting incremented in vRule stats:
   IAE_MSG_APPALIVE -->    Num:   0    MaxSize:    0   MinSize:    0    AvgSize:   0    NoCtx:    0  
   1. If appalive message is not incremented, the instrumentation may not be successful. Check the probe logs for further evaluation
2. In case appalive counters are incremented but the application does not move to Normal, the information in appalive message may not be correct. Enable message log in vRule to view this information. vRule IAE message log contains the appalivemessage received from the probe.
   1. A sample message is depicted below:

      ts=23:53:50.989] [VRE-RCV-HLP0] [DEBUG] [seqno=0] [msg=IAE_MSG_APPALIVE] msgType: IAE_MSG_APPALIVE

      appalive {

        appalive: "{\n    \"appCollectiveId\": 2,\n    \"asiId\": 1409803542,\n    \"health\": 1,\n    \"metadata\": {\n        \"processId\": 19508\n    },\n     \"msgTxnID\": 37528,\n     \"msgType\": 0,\n     \"namespace\": \"1\",\n    \"originTimeStamp\": \"2022-05- 6T20:53:50.989+03:00\",\n     \"sourceVSPComp\": \"Dot Net\",\n    \"targetVSPComp\": \"CMS\",\n     \"version\": \"v1.0\"\n}"

3. In other scenarios, the Kafka connection may not be set up with CMS. vRule waits for the Kafka topic information from vsp-manager before establishing connection with CMS. vRule state log file contains information received from vsp-manager. Verify if the Kafka server information received from vsp-manager is correct. A sample log is provided below:
   [ts=05:00:02.830] [    VRE-RCV-SHM] [ INFO] [seqno=] [uuid=] [msg=] [ns=] [db=] [filter=] [{"version": "v1.0", "sourceVSPComp": "VSP-MANAGER", "feedbackReq": 0, "asiID": 1359585006, "kafkaIP": "10.103.222.145:9092", "groupID": "a3816bef-bcd5-459e-b92a-91daaa55fdaa", "partitionNum": 0, "replicationFactor": 1, "kafkaTopics": {"appAliveKafkaTopic": "VSP_APP_ALIVE", "webIncidentTopic": "VSP_WEB_INCIDENT", "appLogTopic": "VSP_APP_LOG", "hostAppStatsTopic": "VSP_HOST_APP_STATS"}, "targetVSPComp": "AE", "msgType": 7, "msgTxnID": 14}]

Incidents are not Reported to CMS

There may be two reasons, if vRule has started successfully:

1. The required vulnerability may not be configured in CMS
2. Check if Kafka connection is set up properly