============================================================================ ,,_ ____ _ _ ___ o" )~ / ___| _ __ ___ _ __| |_ / \ |_ _| '''' \___ \| '_ \ / _ \| '__| __| / _ \ | | ___) | | | | (_) | | | |_ / ___ \ | | |____/|_| |_|\___/|_| \__| /_/ \_\___| _ __ _ __ ___ _ __ _ __ ___ ___ ___ ___ ___ ___ _ __ | '_ \| '__/ _ \ '_ \| '__/ _ \ / __/ _ \/ __/ __|/ _ \| '__| | |_) | | | __/ |_) | | | (_) | (_| __/\__ \__ \ (_) | | | .__/|_| \___| .__/|_| \___/ \___\___||___/___/\___/|_| |_| |_| ~ A REALLY smart preprocessor module for Snort ~ by BlackLight , http://0x00.ath.cx ============================================================================ This document describes the AI preprocessor module for Snort. It also describes how to get it, install it, configure it and use it correctly. Table of contents: 1. What's Snort AI preprocessor 2. How to get Snort AI preprocessor 3. Installation 3.1 Dependancies 3.2 Configure options 4. Basic configuration 5. Correlation rules 6. Additional documentation =============================== 1. What's Snort AI preprocessor =============================== Snort AI preprocessor is a preprocessor module for Snort whose purpose is making the reading of Snort's alerts more comfortable, clustering false positive alarms emphasizing their root cause in order to reduce log pollution, clustering similar alerts in function of the type and hierarchies over IP addresses and ports that can be decided by the user, depending on the kind of traffic and topology of the network, and constructing the flows of a multi-step attack in function of correlation rules between hyperalerts provided by the developer itself, by third parts or created by the user itself, again, in function of the scenario of the network. It will furthermore possible, in a close future, to correlate the hyperalerts automatically, by self-learning on the base of the acquired alerts. =================================== 2. How to get Snort AI preprocessor =================================== It it strongly suggested to get the latest and always-fresh release of Snort AI preprocessor from GitHub -> http://github.com/BlackLight/Snort_AIPreproc git clone git://github.com/BlackLight/Snort_AIPreproc.git If git is not available on the machine or cannot be used, from the same page you can also choose "download source" and download the source code in tar.gz format. =============== 3. Installation =============== The installation procedure is the usual one: $ ./configure $ make $ make install If you did not install Snort in /usr directory you may need to use the --prefix option with configure for selecting the directory where you installed Snort (for example ./configure --prefix=$HOME/local/snort). If the prefix was specified correctly, and it actually points to the location where Snort was installed, the module binaries should be placed in $SNORT_DIR/lib/snort_dynamicpreprocessor after the installation, and automatically loaded by Snort at the next start. Moreover, a new directory named corr_rules will be created, in /etc/snort if the prefix was /usr or in $SNORT_DIR/etc otherwise, containing XML files describing default correlation rules provided by the developer. This set can be enriched in any moment with new XML files, provided by third parts or created by the user itself, describing more hyperalerts. ================ 3.1 Dependancies ================ Dependancies required for a correct compilation and configuration: - pthread (REQUIRED), used for running multiple threads inside of the module. On a Debian-based system, install libpthread-dev if you don't already have it. - libxml2 (REQUIRED), used for parsing XML files from corr_rules directory. On a Debian-based system, install libxml2-dev if you don't already have it. - libgraphviz (RECOMMANDED), used for generating PNG (and in future PS too) files representing hyperalert correlation graphs from .dot files generated from the software. You can remove this dependancy from the compilation process by specifying --without-graphviz to ./configure, but in this case you will have .dot files, not easily understandable by a human, for representing correlation graphs, and you may need an external graph rendering software for converting them in a more easily readable format. On a Debian system, install libgraphviz-dev if you don't already have it. - libmysqlclient (OPTIONAL), used if you want to read alerts information saved on MySQL DBMS, or enable MySQL support in the module. This option is disabled by default (if not specified otherwise, the module will read the alerts from Snort plain log files), and can be enabled by specifying the option --with-mysql to ./configure. On a Debian-based system you may need to install libmysqlclient-dev. - libpq (OPTIONAL), used if you want to read alerts information saved on PostgreSQL DBMS, or enable PostgreSQL support in the module. This option is disabled by the default, and can be enabled by specifying the option --with-postgresql to ./configure. On a Debian-based system you may need to install libpq-dev. ===================== 3.2 Configure options ===================== You can pass the following options to ./configure script before compiling: --with-mysql - Enables MySQL DBMS support into the module (it requires libmysqlclient) --with-pq - Enables PostgreSQL DBMS support into the module (it requires libpq) --without-graphviz - Disables Graphviz support from the module, avoiding the generation of PNG or PS files representing hyperalerts correlation as well ====================== 4. Basic configuration ====================== After installing the module in Snort installation directory a configuration for this is required in snort.conf. A sample configuration may appear like the following: preprocessor ai: \ hashtable_cleanup_interval 300 \ tcp_stream_expire_interval 300 \ alertfile "/home/youruser/local/snort/log/alert" \ alert_clustering_interval 300 \ correlation_graph_interval 300 \ correlation_rules_dir "/your/snort/dir/etc/corr_rules" \ correlated_alerts_dir "/your/snort/dir/log/correlated_alerts" \ correlation_threshold_coefficient 0.5 \ database ( type="mysql", name="snort", user="snortusr", password="snortpass", host="dbhost" ) \ database_parsing_interval 30 \ clusterfile "/your/snort/dir/log/clustered_alerts" \ cluster ( class="dst_port", name="privileged_ports", range="1-1023" ) \ cluster ( class="dst_port", name="unprivileged_ports", range="1024-65535" ) \ cluster ( class="src_addr", name="local_net", range="192.168.1.0/24" ) \ cluster ( class="src_addr", name="dmz_net", range="155.185.0.0/16" ) \ cluster ( class="src_addr", name="vpn_net", range="10.8.0.0/24" ) \ cluster ( class="dst_addr", name="local_net", range="192.168.1.0/24" ) \ cluster ( class="dst_addr", name="dmz_net", range="155.185.0.0/16" ) \ cluster ( class="dst_addr", name="vpn_net", range="10.8.0.0/24" ) The options are the following: - hashtable_cleanup_interval: The interval that should occur from the cleanup of the hashtable of TCP streams and the next one (default if not specified: 300 seconds) - tcp_stream_expire_interval: The interval that should occur for marking a TCP stream as "expired", if no more packets are received inside of that and it's not "marked" as suspicious (default if not specified: 300 seconds) - alertfile: The file where Snort saves its alerts, if they are saved to a file and not to a database (default if not specified: /var/log/snort/alert) - alert_clustering_interval: The interval that should occur from the clustering of the alerts in the log according to the provided clustering hierarchies and the next one (default if not specified: 300 seconds) - correlation_graph_interval: The interval that should occur from the building of the correlation graph between the clustered alerts and the next one (default if not specified: 300 seconds) - correlation_rules_dir: Directory where the correlation rules are saved, as XML files (default if not specified: /etc/snort/corr_rules) - correlated_alerts_dir: Directory where the information between correlated alerts will be saved, as .dot files ready to be rendered as graphs and, if libgraphviz support is enabled, as .png and .ps files as well (default if not specified: /var/log/snort/clustered_alerts) - correlation_threshold_coefficient: The threshold the software uses for stating two alerts are correlated is avg(correlation coefficient) + k * std_deviation(correlation_coefficient). The value of k is specified through this option, whose value is 0.5 by default, and is dependant on how "sensible" you want the correlation algorithm. A value of k=0 means "consider all the couples of alerts whose correlation coefficient is greater than the average one as correlated" (negative values of k are allowed as well, but unless you know what you're doing they're unrecommended, as some correlation constraints may appear where no correlation exists). When the value of k raises also the threshold for two alerts for being considered as correlated raises. A high value of k may just lead to an empty correlation graph - database: If Snort saves its alerts to a database and the module was compiled with database support (e.g. --with-mysql) this option specifies the information for accessing that database. The fields in side are -- type: DBMS to be used (so far MySQL and PostgreSQL are supported) -- name: Database name -- user: Username for accessing the database -- password: Password for accessing the database -- host: Host holding the database - database_parsing_interval: The interval that should occur between a read of the alerts from database and the next one (default if not specified: 30 seconds) - clusterfile: File where the clustered alerts will be saved by the module (default if not specified: /var/log/snort/clustered_alerts) - cluster: Clustering hierarchy or list of hierarchies to be applied for grouping similar alerts. This option needs to specify: -- class: Class of the cluster node. It may be src_addr, dst_addr, src_port or dst_port -- name: Name for the clustering node -- range: Range of the clustering node. It can include a single port or IP address, an IP range (specified as subnet x.x.x.x/x), or a port range (specified as xxx-xxx) ==================== 5. Correlation rules ==================== The hyperalert correlation rules are specified in $SNORT_DIR/etc/corr_rules directory through a very simple XML syntax, and any user can add some new ones. The files in there must be named like the Snort alert ID they want to model. For example, if we want to model a TCP portscan alert (Snort ID: 122.1.0) as a hyperalert, i.e. as an alert with pre-conditions and post-conditions to be correlated to other alerts, then we need to create a file named 122-1-0.xml inside $SNORT_DIR/etc/corr_rules with a content like the following: 122.1.0 (portscan) TCP Portscan 
HostExists(+DST_ADDR+)
HasService(+DST_ADDR+, +ANY_PORT+) The tag is optional, same for 
  and  if an alert has no
pre-conditions and/or post-conditions, while the  tag is mandatory for
identifying the hyperalert. In this case we say that the pre-condition for a TCP
portscan  for  being  successful  is  that the host +DST_ADDR+ exists (the macro
 +DST_ADDR+   will   automatically   be  expanded  at  runtime  and  substituted
 with   the   target   address   of  the  portscan).  The  post-condition  of  a
portscan  consists  in  the  attacker  knowing that +DST_ADDR+ runs a service on
+ANY_PORT+  (+ANY_PORT+  is another macro that will be expanded on runtime). The
hyperalerts  model  in  corr_rules  are  the knowledge base used for correlating
alerts triggered by Snort, the more information it has inside, the more accurate
and  complete  the  correlation will be. The macros recognized and automatically
expanded           from           these          XML          files          are


- +SRC_ADDR+: The IP address triggering the alert
- +DST_ADDR+: The target IP address in the alert
- +SRC_PORT+: The port from which the alert was triggered
- +DST_PORT+: The port on which the alert was triggered
- +ANY_ADDR+: Identifies any IP address
- +ANY_PORT+: Identifies any port


The correlation between two alerts A and B is made comparing the post-conditions
of  A  with  the  pre-conditions of B. If the correlaton coefficient computed in
this  way  is  greater  than  a  certain  threshold (see "Basic configuration ->
 correlation_threshold_coefficient")    then    the   alerts   are   marked   as
correlated, i.e. the alert A determines the alert B. This supports an elementary
reasoning  algorithm  for  doing inferences on the conditions. For example, if A
has  the  post-condition  "HasService(+DST_ADDR+,  +ANY_PORT+)"  and  B  has the
pre-condition "HasService(+DST_ADDR, 22)", a match between A and B is triggered.
Same   if   A   has   "HostExists(10.8.0.0/24)"  as  post-condition  and  B  has
"HostExists(10.8.0.1)" as pre-condition.

There  is  no  fixed  semantics  for  the  the  predicates in pre-conditions and
post-conditions,  any  predicates  can  be used. The only constraint is that the
same  predicates have the same semantic and prototype in all of the hyperalerts.
For  example,  if  HasService  has a prototype like "HasService(ip_addr, port)",
then   all    the  hyperalerts  should  follow  this  prototype,  otherwise  the
matching    would   fail.   Any  new  predicates  can  be  defined  as  well  in
hyperalerts,      provided     that     it     respects     this     constraint.


===========================
6. Additional documentation
===========================

The additional documentation over the code, functions and data structures can
be automatically generated by Doxygen by typing `make doc', and installed  in
SNORT_DIR/doc then typing `make doc-install'.