<< PREVIOUS NEXT >>

 

 

TROUBLESHOOTING  – vRULE ENGINE

 

vRule requires JDK 17 on the platform. This section lists common issues encountered, possible reasons and recommended workarounds.

 

LOG FILES LOCATION

 

vRule Engine maintains four log files:

  1. Vrule_log.yyyy-mm-dd.n.log - For vRule state logs

  2. Vrule_incident.yyyy-mm-dd.n.log – To capture all incidents generated by vRule in debug mode

  3. Vrule_iaemsg.yyyy-mm-dd.n.log – To capture all message logs received from probe

  4. Vrule_vcr.yyyy-mm-dd.n.log – To capture VCR message debug logs

All the logs are set to info mode. Debug logs are available only when logging is enabled in debug mode. Utilize the commands below to configure the debug mode:

  1. vRule state logs:

    1. vsp-cli config ae edit logLevel debug

       

  2. vRule Incident logs:

    1. vsp-cli config ae edit logReport debug

       

  3. IAE Message log:

    1. vsp-cli config ae edit logMessage debug

       

  4. VCR Message log:

    1. vsp-cli config ae edit logVcr debug

       

 

vRULE DOES NOT START

 

Symptom:

  1. The command “vsp-cli list” lists AE process after application provisioning is successful. If vRule is not listed, AE/vRule has not started successfully

Possible Causes:

  1. JDK may not be installed in the system and vRule may not start

  2. The customer platform may not have write permissions to the tmp directory needed by java

 

Debugging Steps:

  1. Check and verify if JDK 17 is installed. Install JDK it if not already installed

     

    NOTE:  

    Internet connection is required for JDK installation

     

  2. Execute the command on the Linux platform to check the JDK path:

    1. echo $VISREC_JAVA_HOME

       

    2. If the path is not set to the correct directory, add below line in the file /etc/environment

      1. export VIRSEC_JAVA_HOME=<JDK_PATH>

         

    3. Example:

      export VIRSEC_JAVA_HOME=/usr/lib/jvm/openjdk-11-manual-installation/

       

  3. On Windows platform, check if the environment variable VISREC_JAVA_HOME is defined

  4. On Linux, vRule can be manually started using the commands:

    1. cd /opt/virsec/script

       

    2. ./vrule_run.sh

       

    3. Verify that there are no errors in the newly created log files in the directory: /var/virsec/log/vrule/

  5. If vrule is still not starting and vRule log file contains the error “Could not initialize class com.sun.jna.Native”:

    1. Linux: Modify the file /opt/virsec/scripts/vrule_run.sh to add the below option (if it is not already present)

      $VIRSEC_JAVA_HOME/bin/java -server -XX:+UseG1GC

      XX:InitialHeapSize=$VRE_MEM -XX:MaxHeapSize=$VRE_MEM -

      XX:MaxGCPauseMillis=300 -

      DVRE_MEM=$VRE_MEM -

      Djava.io.tmpdir=/opt/virsec/tmp -DNUM_VRULE_WORKER=$NUM_VRULE_WORKER -DSTATS_PATH=/vspstats/vrule/ -

      Djava.library.path=/opt/virsec/lib/ --add-opens

      java.base/jdk.internal.misc=ALL-UNNAMED-

      Dio.netty.tryReflectionSetAccessible=true -

      DDB_CACHE_PATH=/opt/virsec/data/vrule/compiled_db/ -

      DVRULE_CONFIG_PATH=/opt/virsec/data/vrule/ -

      DVRULE_JSON_CONFIG_PATH=/opt/virsec/config/ -

      Dlogback.configurationFile=/opt/virsec/data/vrule/logback.xml -

      DVIRSEC_VSP_SKU_TYPE=$VIRSEC_VSP_SKU_TYPE -

      jar /opt/virsec/bin/vrules_engine-1.0.jar

       

    2. Windows: Modify the file: C:\Program Files (x86)\virsec\vrule\scripts\vrule_run.bat to add the below option (if it is not already present)

       

      set VO8=-Djava.io.tmpdir="%VSP_HOME%\tmp"

      "%VIRSEC_JAVA_HOME%\bin\java" -server -XX:+UseG1GC -

      XX:InitialHeapSize=%VRE_MEM% -

      XX:MaxHeapSize=%VRE_MEM% -XX:MaxGCPauseMillis=300 -DVRE_MEM=%VRE_MEM% -DNUM_VRULE_WORKER=%NUM_VRULE_WORKER% --add-opens

      java.base/jdk.internal.misc=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true %VO8% %VO7% %VO3% %VO1% %VO2% %VO4% %VO5% %V06% -jar "%VSP_HOME%\vrule\bin\vrules_engine-1.0.jar"

       

  6. Check if Host Monitoring is enabled in Protect mode and if any HMM incident is generated with java (vRule process) getting killed. In such cases, add the appropriate exclusion rules in CMS

  7. Run the command below to check if vRule process is running:

    1. ps aux | grep vrules

       

 

WEB APPLICATION DOES NOT MOVE TO NORMAL

 

If vRule process is running but the application does not move to Normal, follow the steps in this section to debug the scenario:

 

  1. Check if appalive counters are getting incremented in vRule stats:

    image55

    1. If appalive message is not incremented, the instrumentation may not be successful. Check the probe logs for further evaluation

  2. In case appalive counters are incremented but the application does not move to Normal, the information in appalive message may not be correct. Enable message log in vRule to view this information. vRule IAE message log contains the appalive message received from the probe.

    1. A sample message is depicted below:

      image56

  3. In other scenarios, the Kafka connection may not be set up with CMS. vRule waits for the Kafka topic information from vsp-manager before establishing connection with CMS. vRule state log file contains information received from vsp-manager. Verify if the Kafka server information received from vsp-manager is correct. A sample log is provided below:

    image57

  4. This information is received and logged during the start of vRule process itself. If this log is not found, vsp-manager log must be analyzed for more information

 

INCIDENTS ARE NOT REPORTED TO CMS

 

There may be two reasons, if vRule has started successfully:

  1. The required vulnerability may not be configured in CMS

  2. Check if Kafka connection is set up properly

 

vRULE COUNTERS AND STATISTICS

 

vRule maintains several counters and statistics that get updated based on the configured rate (Default: 5 seconds)

  1. vRule Engine provides sufficient statistics for all messages it receives:

    1. vsp-cli stats ae reset

       

    2. vsp-cli stats ae view

       

  2. Worker Thread Statistics – it maintains counters for every worker thread, regarding number of incidents generated at each level (attack/threat/log/benign and exception) for every feature

    image58

  3. All the counter types are arranged in rows, Each counter type may have more than one counter arranged under a different column.

  4. Example 1: virsec_Wrk1 sqlfeature - indicates SQLi incidents generated by worker thread 1 for namespace virsec that implements CVE namespace rules

    image59

  5. Example 2: custom_block_Wrk0 sqlfeat - indicates counters for incidents generated by worker thread 0, for namespace custom_block that implements custom deny rules

  6. Receiver counters - maintained by data packets receiver (either SHM-based or RSocket-based for remote vRule)

    image60

  7. RcvdMsg - number of messages received by vRule. The messages consist of all data packets such as HTTP request/response, appalive

    1. Schd Msg - number of messages scheduled for processing by vRule worker threads

    2. Drop - number of packets dropped due to congestion. An increase in this counter over a period of time indicates that the number of configured worker threads is not enough to cater to the peak traffic

    3. Blocked – number of times a worker thread was in congestion. If this is non-zero and keeps increasing, it indicates that the peak data rate is high and number of worker threads must be increased. If this value keeps increasing, examine the flow control counters mentioned in point 10 below

    4. Unblocked - number of times a worker thread recovered from congestion

    5. Rate (MB/s) - live rate at which vRule Engine receives data traffic

    6. AvgSize - average size of received packets

    7. Pending - number of pending messages that the worker thread needs to process

  8. Message counters - Maintain a count of each message type along with min/max and avg size, as depicted below

    image61

  9. The message reception depends on the type of vulnerability enabled in the system. At minimum, IAE_MSG_HTTP_REQ and IAE_MSG_HTTP_RESPONSE counters gets incremented

    1. If CMDi is enabled, then IAE_MSG_CMD gets incremented

    2. If SQLi is enabled, then IAE_MSG_SQL gets incremented

  10. Flow control counters

    image62

    1. Autoscaling - indicates whether autoscaling is enabled or not. “0” – disabled, “1” – enabled. The number of worker thread are automatically scaled up to required number. This requires vRule process restart

      1. If autoscaling is not enabled, enable it using the command:

        vsp-cli config ae edit autoscaling true

         

    2. ReqdWorker - recommends the number of worker thread needed by vRule Engine. This is relevant only if vRule has moved into congestion mode (Blocked in receive counter > 0). A non-zero value indicates that during peak traffic, vRule had gone into congestion (drops > 0). ReqdWorker is the measured number of worker threads needed to handle peak traffic

      1. vsp-cli config ae edit numWorker <number>

         

    3. BRate(MB/s) - peak data rate measured during congestion

 

VSP-CLI COMMANDS FOR vRULE

 

VSP-cli commands that can be used for vRule Engine are:

  1. View the statistics: vRule statistics are located in the file: /vspstats/vrule/vrule.stats that can be watched for live updates of these counters

    1. vsp-cli stats ae view

       

  2. Reset the statistics:

    1. vsp-cli stats ae reset

       

  3. vRule Engine has multiple worker threads running to process the messages in parallel. The commands related to worker threads:

    1. Modify the number of worker threads. vRule Engine must be restarted after modification

      1. vsp-cli config ae edit numWorker <number>

         

    2. Autoscaling can be configured as true or false. If set to true, vRule Engine automatically scales up the number of worker threads

      1. vsp-cli config ae edit autoscaling <true/false>

         

 

<< PREVIOUS NEXT >>