Merge pull request #903 from dedemorton/gs_fixes_1.1_windows

Add changes for 891, 893, and some misc consistency issues

Author: tsg

filebeat/docs/command-line.asciidoc

@@ -4,7 +4,8 @@
 Filebeat does not have any Filebeat-specific command line options. Instead, you
 configure the behaviour of Filebeat by specifying options in the configuration file.
 
-The following command line options from libbeat are also available for Filebeat.
+The following command line options from libbeat are also available for Filebeat. To
+use these options, you need to start Filebeat in the foreground.
 
 To start Filebeat, you must use the `-c config/path` option to specify the path to the
 configuration file.

filebeat/docs/getting-started.asciidoc

@@ -79,8 +79,12 @@ see <<filebeat-configuration-details>>.
 [[filebeat-configuration]]
 === Configuring Filebeat
 
-To configure Filebeat, you edit the _filebeat.yml_ file. Here is a sample of
-`filebeat` section of the _filebeat.yml_ file:
+To configure Filebeat, you edit the configuration file. For rpm and deb, you'll
+find the configuration file at `/etc/filebeat/filebeat.yml`. For mac and win, look in
+the archive that you just extracted.
+
+Here is a sample of the `filebeat` section of the `filebeat.yml` file. Filebeat uses predefined
+default values for most configuration options. 
 
 [source,yaml]
 -------------------------------------------------------------------------------------
@@ -93,7 +97,7 @@ filebeat:
       # For each file found under this path, a harvester is started.
       paths:
         - "/var/log/*.log"
-      # - c:\programdata\elasticsearch\logs\*
+        #- c:\programdata\elasticsearch\logs\*
 
       # Type of the files. Based on this the way the file is read is decided.
       # The different types cannot be mixed in one prospector
@@ -104,9 +108,12 @@ filebeat:
       input_type: log
 -------------------------------------------------------------------------------------
 
-Filebeat uses predefined default values for most configuration options. For the most basic
-Filebeat configuration, you can define a single prospector with a single path. For example:
+To configure Filebeat:
 
+. Define the path (or paths) to your log files.
++
+For the most basic Filebeat configuration, you can define a single prospector with a single path. For example:
++
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat:
@@ -115,22 +122,22 @@ filebeat:
       paths:
         - "/var/log/*.log"
 -------------------------------------------------------------------------------------
-
++
 The prospector in this example harvests all files in the path `/var/log/*.log`, which means
 that Filebeat will harvest all files in the directory `/var/log/` that end with `.log`. All patterns supported
 by https://golang.org/pkg/path/filepath/#Glob[Golang Glob] are also supported here.
-
++
 To fetch all files from a predefined level of subdirectories, the following pattern can be used:
 `/var/log/*/*.log`. This fetches all `.log` files from the subfolders of `/var/log`. It does not
 fetch log files from the `/var/log` folder itself. Currently it is not possible to recursively
 fetch all files in all subdirectories of a directory.
-
++
 A config file can contain multiple prospectors and multiple paths per prospector
 as shown in the following example.
-
++
 NOTE: Make sure a file is not defined more than once across all prospectors because this can lead
 to unexpected behaviour.
-
++
 [source,yaml]
 -------------------------------------------------------------------------------------
 filebeat:
@@ -141,15 +148,31 @@ filebeat:
         - /var/log/wifi.log
     -
       paths:
-        - "/var/log/apache/*"
+        - "/var/log/apache2/*"
 -------------------------------------------------------------------------------------
-
++
 The config file in the example starts two prospectors. The first prospector has two harvesters,
 one harvesting the `system.log` file, and the other harvesting `wifi.log`. The second prospector
 starts a harvester for each file in the apache directory.
 
+. If you are sending output to Elasticsearch, set the IP address and port where Filebeat can find the Elasticsearch installation:
++
+[source,yaml]
+----------------------------------------------------------------------
+# Configure what outputs to use when sending the data collected by the beat.
+# Multiple outputs may be used.
+output:
+  ### Elasticsearch as output
+  elasticsearch:
+    # Array of hosts to connect to.
+     hosts: ["192.168.1.42:9200"]
+----------------------------------------------------------------------
++
+If you are sending output to Logstash, see <<config-filebeat-logstash>> instead.
+
 See <<filebeat-configuration-details>> for more details about each configuration option.
 
+[[config-filebeat-logstash]]
 === Configuring Filebeat to Use Logstash
 
 include::../../libbeat/docs/shared-logstash-config.asciidoc[]

filebeat/docs/migration.asciidoc

@@ -128,7 +128,7 @@ filebeat:
       document_type: stdin
     -
       paths:
-        - "/var/log/apache/httpd-*.log"
+        - "/var/log/apache2/httpd-*.log"
       document_type: apache
 -------------------------------------------------------------------------------------
 
@@ -190,26 +190,23 @@ The equivalent in Filebeat would look like this:
 -------------------------------------------------------------------------------------
 output:
   logstash:
-    enabled: true
-
-    # The list of downstream Logstash servers. <1>
+    # The Logstash hosts. <1>
     hosts:
       - localhost:5043
+      
+    # Network timeout in seconds.
+    timeout: 15
 
     tls: <2>
-      # The path to your SSL client certificate.
-      certificate: ./logstash-forwarder.crt
-
-      # The path to your SSL client certificate key.
-      certificate_key: ./logstash-forwarder.key
-
-      # The path to your trusted SSL CA file. This is used
-      # to authenticate your downstream server.
-      certificate_authorities:
+      # List of root certificates for HTTPS server verifications
+      certificate_authorities: 
         - ./logstash-forwarder.crt
+        
+      # Certificate for TLS client authentication
+      certificate: ./logstash-forwarder.crt
 
-      # Network timeout in seconds.
-      timeout: 15
+      # Client Certificate Key
+      certificate_key: ./logstash-forwarder.key    
 -------------------------------------------------------------------------------------
 
 <1> When multiple hosts are defined, the default behavior in Filebeat is to
@@ -285,7 +282,6 @@ filebeat:
         service: test01
 output:
   elasticsearch:
-    enabled: true
     hosts: ["http://localhost:5043"]
 -------------------------------------------------------------------------------------
 

libbeat/docs/command-line.asciidoc

@@ -1,7 +1,8 @@
 [[command-line-options]]
 == Command Line Options
 
-The following command line options are available for all Beats. For additional
-command line options, see the documentation for your Beat.
+The following command line options are available for all Beats. To use these
+options, you need to start the Beat in the foreground. For additional command
+line options, see the documentation for your Beat.
 
 include::./shared-command-line.asciidoc[]

libbeat/docs/gettingstarted.asciidoc

@@ -110,18 +110,17 @@ You should see a response similar to this:
 [source,shell]
 ----------------------------------------------------------------------
 {
-  "name" : "Banshee",
+  "name" : "Angela Del Toro",
   "cluster_name" : "elasticsearch",
   "version" : {
     "number" : "2.2.0",
-    "build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87",
-    "build_timestamp" : "2015-11-18T22:40:03Z",
+    "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
+    "build_timestamp" : "2016-01-27T13:32:39Z",
     "build_snapshot" : false,
-    "lucene_version" : "5.3.1"
+    "lucene_version" : "5.4.1"
   },
   "tagline" : "You Know, for Search"
 }
-
 ----------------------------------------------------------------------
 
 [[logstash-installation]]
@@ -403,12 +402,11 @@ Use the following commands to run the script:
 curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-{Dashboards-version}.zip
 unzip beats-dashboards-{Dashboards-version}.zip
 cd beats-dashboards-{Dashboards-version}/
-./load.sh -url http://localhost:9200
+./load.sh
 ----------------------------------------------------------------------
 
-Replace `localhost:9200` with the location where Elasticsearch is running.
-For example, `./load.sh -url http://192.168.33.60:9200`. For more command-line options,
-see <<dashboard-load-options>>.
+By default, the script assumes that you are running Elasticsearch on `127.0.0.1:9200`. Use the `-url` option
+to specify a different location. For example: `./load.sh -url http://192.168.33.60:9200`. To specify a Kibana index pattern or pass in user credentials, see <<dashboard-load-options>>.
 
 *win:*
 
@@ -426,47 +424,41 @@ to download and install PowerShell.
 ["source","sh",subs="attributes,callouts"]
 ----------------------------------------------------------------------
 PS > cd beats-dashboards-{Dashboards-version}
-PS > .\load.ps1 -url http://localhost:9200
+PS > .\load.ps1
 ----------------------------------------------------------------------
 +
-Replace `localhost:9200` with the location where Elasticsearch is running. For example,
-`.\load.ps1 -url http://192.168.33.60:9200`. For more command-line options,
-see <<dashboard-load-options>>.
+By default, the script assumes that you are running Elasticsearch on `127.0.0.1:9200`. Use the `-url` option
+to specify a different location. For example: `.\load.ps1 -url http://192.168.33.60:9200`. To specify a Kibana index pattern or pass in user credentials, see <<dashboard-load-options>>.
 +
-NOTE: If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\load.ps1 -url http://localhost:9200`.
+NOTE: If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\load.ps1 -url http://192.168.33.60:9200`.
 
 [[dashboard-load-options]]
-===== Dashboard Load Options
-
-You can specify the following options when you run the load script (`load.sh` or
-`load.ps1`) to load Kibana dashboards. If you specify no options, the script
-assumes that Elasticsearch is running at `http://localhost:9200`.
-
-[[horizontal]]
-*`-h | -help`*::
-Print the help menu.
-
-*`-l | -url <Elasticsearch_url>`*::
-The URL of the server where Elasticsearch is running. For example,
-`-url http://192.168.33.60:9200`.
-
-*`-u | -user <username>:<password>`*::
-The username and password for authenticating to Elasticsearch using Basic
-Authentication. Specify the username and password separated by a colon
-(for example, `-user admin:secret`). By default no username and password are
-used.
-
-*`-i | -index <index_pattern>`*::
-The Kibana index pattern where you want to save the Beats dashboards,
-visualizations, and index patterns. You can use this option to upload the Beats
-dashboards into multiple Kibana installations. For example: 
+===== Other Dashboard Load Options
+When you load the dashboards, you can also specify:
+
+* The Kibana index pattern where you want to load the dashboards in Elasticsearch. For example:
 +
 ["source","sh",subs="attributes,callouts"]
 ----------------------------------------------------------------------
-./load.sh -url http://localhost:9200 -index .kibana_env1
-./load.sh -url http://localhost:9200 -index .kibana_env2
-./load.sh -url http://localhost:9200 -index .kibana_env3
+./load.sh -url http://test.com:9200 -index .kibana_env
+----------------------------------------------------------------------
+
+* The username and password to use for authentication. There are a few ways to pass in the username and password. For example:
++
+["source","sh",subs="attributes,callouts"]
 ----------------------------------------------------------------------
+./load.sh -url https://xyz.found.io -u user:password <1>
+
+./load.sh -url https://xyz.found.io -u admin:$(cat ~/pass-file) <2> 
+
+./load.sh -url https://user:password@xyz.found.io <3>
+----------------------------------------------------------------------
++
+<1> Specify the username and password separated by a colon.
+<2> Use a file to avoid polluting the bash history with the password.
+<3> Put the username and password in the URL.
+
+These options are also available when you run `.\load.ps` on Windows.
 
 [[view-kibana-dashboards]]
 ==== Opening the Beats Dashboards in Kibana

libbeat/docs/shared-logstash-config.asciidoc

@@ -22,9 +22,8 @@ output:
   logstash:
     hosts: ["127.0.0.1:5044"]
 
-    # configure logstash plugin to loadbalance events between
-    # configured logstash hosts
-    #loadbalance: false
+    # Optional load balance the events between the Logstash hosts
+    #loadbalance: true
 ------------------------------------------------------------------------------
 
 In this configuration, `hosts` specifies the Logstash server and the port (`5044`)

packetbeat/docs/command-line.asciidoc

@@ -1,6 +1,7 @@
 == Command Line Options
 
-The following command line options are available for Packetbeat.
+The following command line options are available for Packetbeat. To use these options,
+you need to start Packetbeat in the foreground.
 
 [source,shell]
 ------------------------------------------------------------------------