rwbag(1)

NAME
rwbag - Build a binary Bag from SiLK Flow records

SYNOPSIS
  rwbag --bag-file=KEY,COUNTER,OUTPUTFILE
        [--bag-file=KEY,COUNTER,OUTPUTFILE ...]
        [{ --pmap-file=PATH | --pmap-file=MAPNAME:PATH }]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--invocation-strip] [--print-filenames] [--copy-input=PATH]
        [--compression-method=COMP_METHOD]
        [--ipv6-policy={ignore,asv4,mix,force,only}]
        [--site-config-file=FILENAME]
        {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

  rwbag --help

  rwbag --legacy-help

  rwbag --version

LEGACY SYNOPSIS

  rwbag [--sip-flows=OUTPUTFILE] [--dip-flows=OUTPUTFILE]
        [--sport-flows=OUTPUTFILE] [--dport-flows=OUTPUTFILE]
        [--proto-flows=OUTPUTFILE] [--sensor-flows=OUTPUTFILE]
        [--input-flows=OUTPUTFILE] [--output-flows=OUTPUTFILE]
        [--nhip-flows=OUTPUTFILE]
        [--sip-packets=OUTPUTFILE] [--dip-packets=OUTPUTFILE]
        [--sport-packets=OUTPUTFILE] [--dport-packets=OUTPUTFILE]
        [--proto-packets=OUTPUTFILE] [--sensor-packets=OUTPUTFILE]
        [--input-packets=OUTPUTFILE] [--output-packets=OUTPUTFILE]
        [--nhip-packets=OUTPUTFILE]
        [--sip-bytes=OUTPUTFILE] [--dip-bytes=OUTPUTFILE]
        [--sport-bytes=OUTPUTFILE] [--dport-bytes=OUTPUTFILE]
        [--proto-bytes=OUTPUTFILE] [--sensor-bytes=OUTPUTFILE]
        [--input-bytes=OUTPUTFILE] [--output-bytes=OUTPUTFILE]
        [--nhip-bytes=OUTPUTFILE]
        [--note-add=TEXT] [--note-file-add=FILE]
        [--print-filenames] [--copy-input=PATH]
        [--compression-method=COMP_METHOD]
        [--ipv6-policy={ignore,asv4,mix,force,only}]
        [--site-config-file=FILENAME]
        {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

DESCRIPTION
rwbag reads SiLK Flow records and builds one or more Bag files. A Bag is similar to a set but each key is associated with a counter. Usually the key is some aspect of a flow record (an IP address, a port, the protocol, et cetera), and the counter is a volume (such as the number of flow records or the sum or bytes or packets) for the flow records that match that key. A Bag file supports a single key field and a single counter field; use the Aggregate Bag tools (e.g., rwaggbag(1)) when the key or counter contains multiple fields.

The --bag-file switch is required and it specifies how to create a Bag file. The argument to the switch names the key field to use for the bag, the counter field, and the location where the bag file is to be written. The switch may be repeated to create multiple Bag files.

rwbag reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. If an input file name ends in ".gz", the file is uncompressed as it is read. When the --xargs switch is provided, rwbag reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

If adding a value to a key would cause the value to overflow the maximum value that Bags support, the key's value will be set to the maximum and processing will continue. In addition, if this is the first value to overflow in this Bag, a warning will be printed to the standard error.

If rwbag runs out of memory, it will exit immediately. The output Bag files will remain behind, each with a size of 0 bytes.

Use rwbagcat(1) to see the contents of a bag. To create a bag from textual input or from an IPset, use rwbagbuild(1). rwbagtool(1) allows you to manipulate binary bag files.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--bag-file=KEY,COUNTER,OUTPUTFILE
    Bin flow records by unique KEY, compute the COUNTER for each bin, and write the result to OUTPUTFILE. The list of available KEY and COUNTER values are given immediately below. OUTPUTFILE is the name of a non-existent file, a named pipe, or the keyword "stdout" or "-" to write the binary Bag to the standard output. Repeat the --bag-file switch to create multiple Bag files in a single pass over the data. Only one OUTPUTFILE may use the standard output. See "LEGACY BAG CREATION SWITCHES" for deprecated methods to create Bag files. This switch or one of legacy equivalents is required. Since SiLK 3.12.0.

    rwbag supports the following names for KEY. The case of KEY is ignored.

    sIPv4
        source IP address, either IPv4 or IPv6

    sIPv6
        source IP address, either IPv4 or IPv6

    dIPv4
        destination IP address, either IPv4 or IPv6

    dIPv6
        destination IP address, either IPv4 or IPv6

    sPort
        source port for TCP or UDP, or equivalent

    dPort
        destination port for TCP or UDP, or equivalent

    protocol
        IP protocol

    packets
        count of packets recorded for this flow record

    bytes
        count of bytes recorded for this flow record

    flags
        bit-wise OR of TCP flags over all packets in the flow

    sTime
        starting time of the flow, in seconds resolution

    duration
        duration of the flow, in seconds resolution

    eTime
        ending time of the flow, in seconds resolution

    sensor
        numeric ID of the sensor where the flow was collected

    input
        router SNMP input interface or vlanId if packing tools were configured to capture it (see sensor.conf(5))

    output
        router SNMP output interface or postVlanId

    nhIPv4
        router next hop IP address, either IPv4 or IPv6

    nhIPv6
        router next hop IP address, either IPv4 or IPv6

    initialFlags
        TCP flags on first packet in the flow

    sessionFlags
        bit-wise OR of TCP flags over all packets except the first in the flow

    attributes
        flow attributes set by the flow generator

    application
        guess as to the content of the flow

    sip-country
        the country code of the source IP address. Uses the mapping file specified by the SILK_COUNTRY_CODES environment variable or the country_codes.pmap mapping file, as described in "FILES". (See also ccfilter(3).) Since SiLK 3.12.0.

    scc an alias for sip-country

    dip-country
        the country code of the destination IP address

    dcc an alias for dip-country

    sip-pmap:MAPNAME
        the value that the source IP address maps to in the mapping file whose map-name is MAPNAME. The type of that prefix map must be IPv4-address or IPv6-address. Use --pmap-file to load the mapping file and optionally set its map-name. Since the MAPNAME must be known when the --bag-file switch is parsed, the --pmap-file switch(es) should precede the --bag-file switch(es).

    dip-pmap:MAPNAME
        the value that the destination IP address maps to in the mapping file whose map-name is MAPNAME. See "sip-pmap:MAPNAME".

    sport-pmap:MAPNAME
        the value that the protocol/source-port pair maps to in the mapping file whose map-name is MAPNAME. The type of that prefix map must be proto-port. Use --pmap-file to load the mapping file and optionally set its map-name. Since the MAPNAME must be known when the --bag-file switch is parsed, the --pmap-file switch(es) should precede the --bag-file switch(es).

    dport-pmap:MAPNAME
        the value that the protocol/destination-port pair maps to in the mapping file whose map-name is MAPNAME. See "sport-pmap:MAPNAME".

    rwbag supports the following names for COUNTER. The case of COUNTER is ignored.

    records
        count of the number of flow records that match the key

    flows
        an alias for records

    sum-packets
        the sum of the packet counts for flow records that match the key

    packets
        an alias for sum-packets

    sum-bytes
        the sum of the byte counts for flow records that match the key

    bytes
        an alias for sum-bytes

--pmap-file=PATH
--pmap-file=MAPNAME:PATH
    Load the the prefix map file from PATH for use when the key part of the argument to the --bag-file switch is one of "sip-pmap", "dip-pmap", "sport-pmap", or "dport-pmap". Specify PATH as "-" or "stdin" to read from the standard input. If MAPNAME is specified, it overrides the map-name contained in the prefix map file itself. If no map-name is available, rwbag exits with an error. The switch may be repeated to load multiple prefix map files; each file must have a unique map-name. To create a prefix map file, use rwpmapbuild(1). Since SiLK 3.12.0.

--note-strip
    Do not copy the notes (annotations) from the input files to the output file(s). When this switch is not specified, notes from the input files are copied to the output. Since SiLK 3.12.2.

--note-add=TEXT
    Add the specified TEXT to the header of every output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of every output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--invocation-strip
    Do not record any command line history: do not copy the invocation history from the input files to the output file(s), and do not record the current command line invocation in the output. The invocation may be viewed with rwfileinfo(1). Since SiLK 3.12.0.

--print-filenames
    Print to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary SiLK Flow records read as input to the specified file or named pipe. PATH may be "stdout" or "-" to write flows to the standard output as long as no Bag file is being written there.

--ipv6-policy=POLICY
    Determine how IPv4 and IPv6 flows are handled when SiLK has been compiled with IPv6 support. When the switch is not provided, the SILK_IPV6_POLICY environment variable is checked for a policy. If it is also unset or contains an invalid policy, the POLICY is mix. When SiLK has not been compiled with IPv6 support, IPv6 flows are always ignored, regardless of the value passed to this switch or in the SILK_IPV6_POLICY variable. The supported values for POLICY are:

    ignore
        Ignore any flow record marked as IPv6, regardless of the IP addresses it contains. Only IP addresses contained in IPv4 flow records will be added to the bag(s).

    asv4
        Convert IPv6 flow records that contain addresses in the ::ffff:0:0/96 netblock (that is, IPv4-mapped IPv6 addresses) to IPv4 and ignore all other IPv6 flow records.

    mix Process the input as a mixture of IPv4 and IPv6 flow records. When creating a bag whose key is an IP address and the input contains IPv6 addresses outside of the ::ffff:0:0/96 netblock, this policy is equivalent to force; otherwise it is equivalent to asv4.

    force
        Convert IPv4 flow records to IPv6, mapping the IPv4 addresses into the ::ffff:0:0/96 netblock.

    only
        Process only flow records that are marked as IPv6. Only IP addresses contained in IPv6 flow records will be added to the bag(s).

    Regardless of the IPv6 policy, when all IPv6 addresses in the bag are in the ::ffff:0:0/96 netblock, rwbag treats them as IPv4 addresses and writes an IPv4 bag. When any other IPv6 addresses are present in the bag, the IPv4 addresses in the bag are mapped into the ::ffff:0:0/96 netblock and rwbag writes an IPv6 bag.

--compression-method=COMP_METHOD
    Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

    none
        Do not compress the output using an external library.

    zlib
        Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.

    snappy
        Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.

    best
        Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available. Only compress the output when writing to a file.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwbag searches for the site configuration file in the locations specified in the "FILES" section.

--xargs
--xargs=FILENAME
    Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwbag opens each named file in turn and reads records from it as if the filenames had been listed on the command line.

--help
    Print the available options and exit.

--legacy-help
    Print help, including legacy switches. See the "LEGACY BAG CREATION SWITCHES" section below for these switches.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

LEGACY BAG CREATION SWITCHES
The following switches are deprecated as of SiLK 3.12.0. These switches may be used in conjunction with the --bag-file switch.

--sip-flows=OUTPUTFILE
    Equivalent to --bag-file=sIPv4,records,OUTPUTFILE. Count number of flows by unique source IP.

--sip-packets=OUTPUTFILE
    Equivalent to --bag-file=sIPv4,sum-packets,OUTPUTFILE. Count number of packets by unique source IP.

--sip-bytes=OUTPUTFILE
    Equivalent to --bag-file=sIPv4,sum-bytes,OUTPUTFILE. Count number of bytes by unique source IP.

--dip-flows=OUTPUTFILE
    Equivalent to --bag-file=dIPv4,records,OUTPUTFILE. Count number of flows by unique destination IP.

--dip-packets=OUTPUTFILE
    Equivalent to --bag-file=dIPv4,sum-packets,OUTPUTFILE. Count number of packets by unique destination IP.

--dip-bytes=OUTPUTFILE
    Equivalent to --bag-file=dIPv4,sum-bytes,OUTPUTFILE. Count number of bytes by unique destination IP.

--sport-flows=OUTPUTFILE
    Equivalent to --bag-file=sPort,records,OUTPUTFILE. Count number of flows by unique source port.

--sport-packets=OUTPUTFILE
    Equivalent to --bag-file=sPort,sum-packets,OUTPUTFILE. Count number of packets by unique source port.

--sport-bytes=OUTPUTFILE
    Equivalent to --bag-file=sPort,sum-bytes,OUTPUTFILE. Count number of bytes by unique source port.

--dport-flows=OUTPUTFILE
    Equivalent to --bag-file=dPort,records,OUTPUTFILE. Count number of flows by unique destination port.

--dport-packets=OUTPUTFILE
    Equivalent to --bag-file=dPort,sum-packets,OUTPUTFILE. Count number of packets by unique destination port.

--dport-bytes=OUTPUTFILE
    Equivalent to --bag-file=dPort,sum-bytes,OUTPUTFILE. Count number of bytes by unique destination port.

--proto-flows=OUTPUTFILE
    Equivalent to --bag-file=protocol,records,OUTPUTFILE. Count number of flows by unique protocol.

--proto-packets=OUTPUTFILE
    Equivalent to --bag-file=protocol,sum-packets,OUTPUTFILE. Count number of packets by unique protocol.

--proto-bytes=OUTPUTFILE
    Equivalent to --bag-file=protocol,sum-bytes,OUTPUTFILE. Count number of bytes by unique protocol.

--sensor-flows=OUTPUTFILE
    Equivalent to --bag-file=sensor,records,OUTPUTFILE. Count number of flows by unique sensor ID.

--sensor-packets=OUTPUTFILE
    Equivalent to --bag-file=sensor,sum-packets,OUTPUTFILE. Count number of packets by unique sensor ID.

--sensor-bytes=OUTPUTFILE
    Equivalent to --bag-file=sensor,sum-bytes,OUTPUTFILE. Count number of bytes by unique sensor ID.

--input-flows=OUTPUTFILE
    Equivalent to --bag-file=input,records,OUTPUTFILE. Count number of flows by unique input interface index.

--input-packets=OUTPUTFILE
    Equivalent to --bag-file=input,sum-packets,OUTPUTFILE. Count number of packets by unique input interface index.

--input-bytes=OUTPUTFILE
    Equivalent to --bag-file=input,sum-bytes,OUTPUTFILE. Count number of bytes by unique input interface index.

--output-flows=OUTPUTFILE
    Equivalent to --bag-file=output,records,OUTPUTFILE. Count number of flows by unique output interface index.

--output-packets=OUTPUTFILE
    Equivalent to --bag-file=output,sum-packets,OUTPUTFILE. Count number of packets by unique output interface index.

--output-bytes=OUTPUTFILE
    Equivalent to --bag-file=output,sum-bytes,OUTPUTFILE. Count number of bytes by unique output interface index.

--nhip-flows=OUTPUTFILE
    Equivalent to --bag-file=nhIPv4,records,OUTPUTFILE. Count number of flows by unique next hop IP.

--nhip-packets=OUTPUTFILE
    Equivalent to --bag-file=nhIPv4,sum-packets,OUTPUTFILE. Count number of packets by unique next hop IP.

--nhip-bytes=OUTPUTFILE
    Equivalent to --bag-file=nhIPv4,sum-bytes,OUTPUTFILE. Count number of bytes by unique next hop IP.

EXAMPLES
In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

  Bag of Protocol:Byte
Read the SiLK Flow file data.rw and create the Bag proto-byte.bag that contains the total byte-count seen for each protocol by using protocol as the key and sum-bytes as the counter:

 $ rwbag --bag-file=protocol,sum-bytes,proto-byte.bag data.rw

Use rwbagcat(1) to view the result:

 $ rwbagcat proto-byte.bag
          1|            10695328|
          6|        120536195111|
         17|            24500079|

Specify the output path as "-" to pass the Bag file from rwbag directly into rwbagcat.

 $ rwbag --bag-file=protocol,sum-bytes,- data.rw    \
   | rwbagcat
          1|            10695328|
          6|        120536195111|
         17|            24500079|

Compare that to this rwuniq(1) command.

 $ rwuniq --field=protocol --value=bytes --sort-output data.rw
 pro|               Bytes|
   1|            10695328|
   6|        120536195111|
  17|            24500079|

One advantage of Bag files over rwuniq is that the data remains in binary form where it can be manipulated by rwbagtool(1).

  Two Bags in a Single Pass
Read records from rwfilter(1) and build Bag files sip-flow.bag and dip-flow.bag that count the number of flows seen for each source address and for each destination address, respectively.

 $ rwfilter ... --pass=stdout                       \
   | rwbag --bag-file=sipv4,records,sip-flow.bag    \
        --bag-file=dipv4,records,dip-flow.bag

  Using a Network Prefix
To create sip16-byte.bag that contains the number of bytes seen for each /16 found in the source address field, use the rwnetmask(1) tool prior to feeding the input to rwbag:

 $ rwfilter ... --pass=stdout                       \
   | rwnetmask --4sip-prefix-length=16              \
   | rwbag --bag-file=sipv4,sum-bytes,sip16-byte.bag

 $ rwbagcat sip16-byte.bag | head -4
        10.4.0.0|               18260|
        10.5.0.0|              536169|
        10.9.0.0|               55386|
       10.11.0.0|             5110438|

To print the IP addresses of an existing Bag into /16 prefixes, use the --network-structure switch of rwbagcat(1).

 $ rwfilter ... --pass=stdout                   \
   | rwbag --bag-file=sipv4,sum-bytes,-         \
   | rwbagcat --network-structure=B             \
   | head -4
        10.4.0.0/16|               18260|
        10.5.0.0/16|              536169|
        10.9.0.0/16|               55386|
       10.11.0.0/16|             5110438|

  Bag of Country Codes
As of SiLK 3.12.0, a Bag file may contain a country code as its key. Create scc-pkt.bag that sums the packet count by country.

 $ rwbag --bag-file=sip-country,sum-packets,scc-pkt.bag
 $ rwbagcat scc-pkt.bag
 --|                 840|
 a1|                 284|
 a2|                   1|
 ae|                   8|

  Bag of Prefix Map Values
rwbag and rwbagbuild(1) can use a prefix map file as the key in a Bag file as of SiLK 3.12.0. For example, to lookup each source address in the prefix map file ip-map.pmap that maps from address to "type of service", use the --pmap-file switch to specify the prefix map file, and specify the Bag's key as "sip-pmap:"map-name, where map-name is either the map-name stored in the prefix map file or a name that is provided as part of the --pmap-file argument. (A prefix map's map-name is available via the rwfileinfo(1) command.)

 $ rwfileinfo --field=prefix-map ip-map.pmap
 ip-map.pmap:
   prefix-map          v1: service-host
 $
 $ rwbag --pmap-file=ip-map.pmap                            \
        --bag-file=sip-pmap:service-host,bytes,srvhost.bag  \
        data.rw

Multiple --pmap-file switches may be specified which may be useful when generating multiple Bag files in a single invocation. On the command line, the --pmap-file switch that defines the map-name must preceded the --bag-file where the map-name is used.

The prefix map file is not stored as part of the Bag, so you must provide the name of the prefix map when running rwbagcat.

 $ rwbagcat srvhost.bag
 rwbagcat: The --pmap-file switch is required for \
         Bags containing sip-pmap keys
 $ rwbagcat --pmap-file=ip-map.pmap srvhost.bag
          external|         59950837766|
          internal|         60602999159|
               ntp|              588316|
               dns|            14404581|
              dhcp|             2560696|

rwbag also has support for prefix map files that map from a protocol-port pair to a label. The proto-port.pmap file does not have a map-name so a name must be provided on the rwbag command line.

 $ rwfileinfo --field=prefix-map proto-port.pmap
 proto-port.pmap:
 $
 $ rwbag --pmap-file=srvport:proto-port.pmap                \
        --bag-file=sip-pmap:srvport,flows,srvport.bag       \
        data.rw
 $ rwbagcat --pmap-file=proto-port.pmap srvport.bag | head -4
      ICMP|               15622|
       UDP|               62216|
   UDP/DNS|               62216|
  UDP/DHCP|               15614|

ENVIRONMENT

SILK_COUNTRY_CODES
    This environment variable allows the user to specify the country code mapping file that rwbag uses when mapping an IP to a country for the "sip-country" and "dip-country" keys. The value may be a complete path or a file relative to the SILK_PATH. See the "FILES" section for standard locations of this file.

SILK_IPV6_POLICY
    This environment variable is used as the value for --ipv6-policy when that switch is not provided.

SILK_CLOBBER
    The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_COMPRESSION_METHOD
    This environment variable is used as the value for --compression-method when that switch is not provided. Since SiLK 3.13.0.

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwbag may use this environment variable when searching for the SiLK site configuration file.

SILK_PATH
    This environment variable gives the root of the install tree. When searching for configuration files, rwbag may use this environment variable. See the "FILES" section for details.

FILES

${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/usr/share/silk/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf
    Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.

$SILK_COUNTRY_CODES
$SILK_PATH/share/silk/country_codes.pmap
$SILK_PATH/share/country_codes.pmap
/usr/share/silk/country_codes.pmap
/usr/share/country_codes.pmap
    Possible locations for the country code mapping file required by the "sip-country" and "dip-country" keys.

SEE ALSO
rwbagbuild(1), rwbagcat(1), rwbagtool(1), rwaggbag(1), rwfileinfo(1), rwfilter(1), rwnetmask(1), rwpmapbuild(1), rwuniq(1), ccfilter(3), sensor.conf(5), silk(7), zlib(3)