Backport: doc changes in #2953, #3002, #3010, #3036, #3041 (#3043)

* Add consulbeat (#2953)

* Add redditbeat to community beats list

* Improve "pipelines" documentation

Mention the similarity to the "indices" array and add an example.

* Default docs update for filbeat

* Default docs for close_eof and close_timeout added
* Remove note for logstash-forward based code
* Updated output docs as elasticsearch can now also be used for parsing (ingest)

* Show download link to dashboards

Author: dedemorton

filebeat/docs/overview.asciidoc

@@ -1,10 +1,8 @@
 [[filebeat-overview]]
 == Overview
 
-Filebeat is a log data shipper initially based on the https://github.com/elastic/logstash-forwarder[Logstash-Forwarder]
-source code. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files,
-and forwards them either to https://www.elastic.co/products/logstash[Logstash] for parsing or directly to
-https://www.elastic.co/products/elasticsearch[Elasticsearch] for indexing.
+Filebeat is a log data shipper. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files,
+and forwards them either to https://www.elastic.co/products/elasticsearch[Elasticsearch] or https://www.elastic.co/products/logstash[Logstash] for indexing.
 
 Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to the spooler, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat.
 

filebeat/docs/reference/configuration/filebeat-options.asciidoc

@@ -233,7 +233,7 @@ WINDOWS:  If your Windows log rotation system shows errors because it can't rota
 
 WARNING: Only use this option if you understand that data loss is a potential side effect.
 
-When this option is enabled, Filebeat closes a file as soon as the end of a file is reached. This is useful when your files are only written once and not updated from time to time. For example, this happens when you are writing every single log event to a new file.
+When this option is enabled, Filebeat closes a file as soon as the end of a file is reached. This is useful when your files are only written once and not updated from time to time. For example, this happens when you are writing every single log event to a new file. This option is disabled by default.
 
 [[close-timeout]]
 ===== close_timeout
@@ -246,6 +246,8 @@ When you use `close_timeout` for logs that contain multiline events, the harvest
 
 The `close_timeout` setting won't apply if your output is stalled and no further events can be sent. At least one event must be sent after `close_timeout` elapses so the harvester can be closed after sending the event.
 
+This option is set to 0 by default which means it is disabled.
+
 
 [[clean-options]]
 ===== clean_*

libbeat/docs/communitybeats.asciidoc

@@ -11,6 +11,7 @@ https://github.com/radoondas/apachebeat[apachebeat]:: Reads status from Apache H
 https://github.com/goomzee/burrowbeat[burrowbeat]:: Monitors Kafka consumer lag using Burrow.
 https://github.com/goomzee/cassandrabeat[cassandrabeat]:: Uses Cassandra's nodetool cfstats utility to monitor Cassandra database nodes and lag.
 https://github.com/aidan-/cloudtrailbeat[cloudtrailbeat]:: Reads events from Amazon Web Services' https://aws.amazon.com/cloudtrail/[CloudTrail].
+https://github.com/Pravoru/consulbeat[consulbeat]:: Reads services health checks from consul and pushes them to elastic.
 https://github.com/Ingensi/dockbeat[dockbeat]:: Reads Docker container
 statistics and indexes them in Elasticsearch.
 https://github.com/radoondas/elasticbeat[elasticbeat]:: Reads status from an Elasticsearch cluster and indexes them in Elasticsearch.
@@ -36,6 +37,7 @@ managers.
 https://github.com/kozlice/phpfpmbeat[phpfpmbeat]:: Reads status from PHP-FPM.
 https://github.com/joshuar/pingbeat[pingbeat]:: Sends ICMP pings to a list
 of targets and stores the round trip time (RTT) in Elasticsearch.
+https://github.com/voigt/redditbeat[redditbeat]:: Collects new Reddit Submissions of one or multiple Subreddits.
 https://github.com/chrsblck/redisbeat[redisbeat]:: Used for Redis monitoring.
 https://github.com/consulthys/retsbeat[retsbeat]:: Collects counts of http://www.reso.org[RETS] resource/class records from https://en.wikipedia.org/wiki/Multiple_listing_service[Multiple Listing Service] (MLS) servers.
 https://github.com/martinhoefling/saltbeat[saltbeat]:: Reads events from salt master event bus.

libbeat/docs/index.asciidoc

@@ -12,6 +12,7 @@ include::./version.asciidoc[]
 :ES-version: {stack-version}
 :LS-version: {stack-version}
 :Kibana-version: {stack-version}
+:dashboards: https://artifacts.elastic.co/downloads/beats/beats-dashboards/beats-dashboards-{stack-version}.zip
 
 include::./overview.asciidoc[]
 

libbeat/docs/newdashboards.asciidoc

@@ -58,7 +58,7 @@ Elasticsearch running on localhost for a single Beat (eg. Metricbeat):
 ./scripts/import_dashboards -file metricbeat-dashboards-1.1.zip
 ----------------------------------------------------------------------
 
-- from the official zip archive available under http://artifacts.elastic.co/:
+- from the official zip archive available at {dashboards}:
 +
 [source,shell]
 ----------------------------------------------------------------------

libbeat/docs/outputconfig.asciidoc

@@ -194,10 +194,35 @@ For more information, see <<configuring-ingest-node>>.
 
 ===== pipelines
 
-Array of pipeline selector configurations supporting conditionals, format string
-based field access and name mappings. The first rule matching will be used to
-set the `pipeline` for the event to be published. If `pipelines` is missing or no
-rule matches, the `pipeline` field will be used.
+Similar to the `indices` array, this is an array of pipeline selector
+configurations supporting conditionals, format string based field access
+and name mappings. The first rule matching will be used to set the
+`pipeline` for the event to be published. If `pipelines` is missing or
+no rule matches, the `pipeline` field will be used.
+
+Example elasticsearch output with `pipelines`:
+
+["source","yaml"]
+------------------------------------------------------------------------------
+filebeat.prospectors:
+- paths: ["/var/log/app/normal/*.log"]
+  fields:
+    type: "normal"
+- paths: ["/var/log/app/critical/*.log"]
+  fields:
+    type: "critical"
+
+output.elasticsearch:
+  hosts: ["http://localhost:9200"]
+  index: "filebeat-%{+yyyy.MM.dd}"
+  pipelines:
+    - pipeline: critical_pipeline
+      when.equals:
+        type: "critical"
+    - pipeline: normal_pipeline
+      when.equals:
+        type: "normal"
+------------------------------------------------------------------------------
 
 ===== template