Web Troubleshooting

Web Protection - Java

File Locations and Configurations

Here are the important file locations

1. Statistics File: 
   1. Windows: <VSP_HOME>/vsp-stats/iae-java/<app-context>-stats
   2. Linux: vspstats/iae-java/<app-context>-stats
2. Log File:
   1. Windows: <VSP_HOME>/log/iae-java/iae-java-<app-context>.log.<data>
   2. Linux: /var/virsec/log/iae-java/ iae-java-<app-context>.log.<data>
3. For disabling instrumentation, following modifications are needed in the file: /vsp-home/iae-java/iae-<app-context>.properties. Once the modifications are complete, ensure that the application is restarted
   1. Disable JDBC instrumentation by adding:
      disableDB=true 
   2. Disable File Event instrumentation by adding:
      disableFileEvent=true 
   3. Disable CMD instrumentation by adding:
      disableCMD=true 
   4. Disable RFI instrumentation by adding:
      disableRFI=true 
   5. Disable stats by modifying:
      enableStats=false 
   6. Disable Threadpool instrumentation by modifying:
      instrumentThreadPool=false 
4. In case of any permission issues or security reasons, modify the logging path and stats path in the file: /vsp-home/iae-java/logging.properties. Restart Application after the modification

Application Status not Normal

1. Check VSP-Web (Java) statistics in the statistics file
   1. A non-zero App Alive message counter indicates that VSP-Web Java is up and running
   2. If the counter is zero, verify that “App Alive attempt failed” counter is increasing while the application is being accessed. If it increases, then the vIPC component is rejecting the app-alive message. Check the status of the vsp-services using vsp-cli utility
   3. If App Alive message counter is increasing, the CMS configuration is not being properly received. Check the component iae-assist
2.  If the statistics file is not present, check the log file. It can be one of the below reasons:
   1. If the log file is not created, verify for any permission errors in the application or server logs
   2. If permission errors are not logged, verify the application instrumentation as below:
      1. Verify using the command <jdk-bin>/jps -lvm. If jps is not present, use absolute path to execute the command
      2. If the application java process does not have virsec arguments, it is not instrumented. 
      3. If virsec agruments are present but the log file is not created, reach out to the Virsec team

Events not Reaching vRule

1. Check if application is in Normal state in CMS. If not, refer VSP-Web Java Application Status not Normal section to find the recommended actions
2. If application is in Normal state, check VSP-Web Java statistics file
   1. Monitor the counters while the application is being accessed.
   2. Check which counters are increasing.
   3. If the counters SENT are increasing, VSP Web Java component is sending events to vIPC. Verify that the vsp-services or vIPC-server status using VSP-CLI utility
   4. If the counters SENT are not increasing but the counter INTERCEPTED is increasing, there might be some issue with VSP Web Java component
   5. Check the logs of VSP Web Java component for any errors. Reach out to the Virsec team with the details of the error
   6. If the counters BadStatus are increasing, vIPC component is rejecting the messages. Verify that the vsp-services or vIPC-server status using VSP-CLI utility
   7. If NON_PROV_REQUEST counter is increasing, it means CMS config data is not available. Check the component iae-assist

Attacks not Detected

1. Check if application is in Normal state in CMS. If not, refer VSP-Web Java Application Status not Normal section to find the recommended actions
2. If application is in Normal State, check VSP-Web Java statistics file
3. The list of vulnerabilities and the respective events are provided in the below table:

   Vulnerability	Events
   RXX, CRLF	HTTP_REQ(threats only), HTTP_RES
   Stored XSS	HTTP_RES
   DOM XSS	DOMXSS_EVENTS
   SQLi	HTTP_REQ, SQL_EVENTS
   Path Traversal	HTTP_REQ, FILE_EVENTS
   Command Injection	HTTP_REQ, CMD_EVENTS
   RFI	HTTP_REQ, RFI_EVENTS
   Classload logging	CLASSLOAD_EVENTS
   Software exception logging	SFWX_EVENTS
   CSRF	CSRF_EVENTS

4. Monitor the counter values when the Application is accessed
5. Verify that there is increment in counters mentioned in the table above for the respective vulnerabilities
6. In cases where the counter values are not varying, search the log files for the vulnerability. A sample log is provided below:
   IAE_2.4 ASI ID: 783040263, Namespace ID: 23, CollectiveId: 22, AppContext: windows2012_atlassian_confluence, Vulnerability Mask: 36671, Protect mode 0, SQLi: true, InsiderProtect: false, CSRF: true, PathTraversal: true, CMDi: true, CRLFi: true, S-XSS: true, Dom-XSS: true, R-XSS: true, LFI: true, LFI-Dir: , LFI-Ext: , RFI: true, RFI-URL: http://google.com, classLoadLog: false, softwareExcLog: true, RunningMapVersion: 3
7. If the counters SENT are increasing, VSP-Web Java is sending events to vIPC. Verify that the vsp-services or vIPC-server status using VSP-cli utility
8. If the counter is not increasing and the counter INTERCEPTED is increasing, there might be some issue with VSP-Web Java
   1. Check the logs for any errors. Reach out to the Virsec team with the details of the error
9. If counters BadStatus are increasing, vIPC is rejecting the messages. Verify that the vsp-services or vIPC-server status using VSP-cli utility

Provisioning Failures

1. Check web-assist is running in the server using vsp-cli command
   1. If it is running, verify startup script path in the cms configuration was given properly. 
   2. Only editable file are supported. 
   3. Apart from setenv.sh and setEnv.bat (only in case of tomcat), if the file is not present in the server, provision will fail. File should exist in the server.
   4. If it was given properly, check web-assist logs and find out any error appearing the logs.
2. If web-assist is not running, check why web-assist was not started

Application status not Normal after vRule configuration change

Symptom: Application is not moving to the normal state in CMS after a configuration change from embedded to remote vRule engine or vice versa 

Recommended Action:  Ensure that both the Business application and VSP services are restarted

Web Protection - Node.js

File Locations and Configurations

Here are the important file locations

Statistics File:  /vsp-stats/iae-nodejs/

Log File: /var/virsec/log/iae_nodejs/iae-nodejs.log

Configuration Files: 
/opt/virsec/iae_nodejs/config/iae.yml
/opt/virsec/iae_nodejs/config/logger.yml

Application Status not Normal

1. Check iae-nodejs statistics in the file: VSP_HOME/vspstats/iae-nodejs/<app-context>-stats
   1. A non-zero App Alive message counter indicates that VSP-Web Node.js is up and running
   2. If the counter is zero, verify that “App Alive attempt failed” counter is increasing while the application is being accessed. If it increases, then the vIPC component is rejecting the app-alive message. Check the status of the VSP-services using VSP-cli utility
   3. If App Alive message counter is increasing, the CMS configuration is not being received correctly. Check the component iae-assist
2. If the statistics file is not present, check the log file. It can be due to one of the below reasons:
   1. If the log file is not created, verify for any permission error in the application or server logs
      1. For permission errors, utilize VSP-CLI utility to update the log file path OR update the path in the file /opt/virsec/iae_nodejs/config/logger.yml
   2. If permission errors are not logged, perform the below steps:
      1. Verify whether the application startup script contains a require (‘/opt/virsec/iae_nodejs’) entry. If this entry exists, it indicates that instrumentation is enabled
      2. If it does not exist, then verify if the correct file path exists for the application start-up script and if write permissions are present for the file
      3. Check the web-assist log file <appcontextname>.log for any errors
   3. Check the file /var/virsec/log/iae_nodejs/iae-nodejs.logsfor the below errors:
      1. Error in iae-assist for job config with the message “IAERR_2.3 Failed to retrieve IAEAssist JobConfig with return status”
      2. For Error in lpc encode library at opening channel with the message “IAERR_4.2 vIPC Client(encode)  Channel is failed to open with return status as”
   4. Check the node process by executing the command:

      ps -ef | grep node 

   5. Check for any error in file /var/virsec/log/iae_nodejs/iae-nodejs.log
   6. If virsec agruments are present but the log file is not created, reach out to the Virsec team

Events not Reaching vRule

1. Check if application is in Normal state in CMS. If not, refer VSP-Web Node.js Application Status not Normal section to find the recommended actions
2. If application is in Normal state, check VSP-Web Node.js statistics file
   1. Monitor the counters while the application is being accessed.
   2. Check which counters are increasing.
   3. If the counters SENT are increasing, VSP Web Node.js component is sending events to vIPC. Verify that the vsp-services or vIPC-server status using VSP-CLI utility
   4. If the counters SENT are not increasing but the counter INTERCEPTED is increasing, there might be some issue with VSP Web Node.js component
   5. Check the logs of VSP Web Node.js component for any errors. Reach out to the Virsec team with the details of the error
   6. If the counters BadStatus are increasing, vIPC component is rejecting the messages. Verify that the vsp-services or vIPC-server status using VSP-CLI utility
   7. If NON_PROV_REQUEST counter is increasing, it means CMS config data is not available. Check the component iae-assist

Attacks not Detected

1. Check if application is in Normal state in CMS. If not, refer VSP-WebNode.js Application Status not Normal section to find the recommended actions
2. If application is in Normal State, check VSP-Web Node.js statistics file
3. The list of vulnerabilities and the respective events are provided in the below table:

   Vulnerability	Events
   RXX, CRLF	HTTP_REQ(threats only), HTTP_RES
   Stored XSS	HTTP_RES
   DOM XSS	DOMXSS_EVENTS
   SQLi	HTTP_REQ, SQL_EVENTS
   Path Traversal	HTTP_REQ, FILE_EVENTS
   Command Injection	HTTP_REQ, CMD_EVENTS
   RFI	HTTP_REQ, RFI_EVENTS
   Classload logging	CLASSLOAD_EVENTS
   Software exception logging	SFWX_EVENTS
   CSRF	CSRF_EVENTS

4. Monitor the counter values when the Application is accessed
5. Verify that the counters mentioned in the above table get incremented for the respective vulnerabilities
6. In cases where the counter values are not varying, search the log files for the vulnerability. A sample log is provided below:
   IAE_2.4  IAEAssist JobConfig
7. If the counters SENT are increasing, VSP-Web Node.js is sending events to vIPC. Verify that the vsp-services or vIPC-server status using VSP-cli utility
8. If the counter is not increasing and the counter INTERCEPTED is increasing, there might be some issue with VSP-Web Node.js
   1. Check the logs for any errors. Reach out to the Virsec team with the details of the error
9. If counters BadStatus are increasing, vIPC is rejecting the messages. Verify that the vsp-services or vIPC-server status using VSP-cli utility

Provisioning Failures

1. Verify that the web-assist is running in the server using vsp-cli command
   1. If it is running, verify that the provisioning scripts are executed correctly 
   2. If the scripts are executed correctly, check the web-assist logs for any errors
2. If web-assist is not running, verify the logs for more information on the startup failure