Skip to content

Commit 9e6358c

Browse files
authored
Merge pull request #2724 from tsg/backport_2631
Backport 2631: Breaking changes guide + fixes
2 parents 2be695c + 45fdb9a commit 9e6358c

File tree

3 files changed

+318
-0
lines changed

3 files changed

+318
-0
lines changed

libbeat/docs/breaking.asciidoc

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
[[breaking-changes]]
2+
== Breaking changes
3+
4+
As a general rule, we strive to keep backwards compatibility between minor
5+
versions (e.g. 5.x to 5.y) so you can upgrade without any configuration file
6+
changes, but there are breaking changes between major versions (e.g. 5.x to
7+
6.y).
8+
9+
We list the breaking changes at the top of the <<release-notes,release notes>>
10+
for each version.
11+
12+
[[breaking-changes-5.0]]
13+
=== Breaking changes in 5.0
14+
15+
This section discusses the changes that you should be aware of if you upgrade
16+
the Beats from version 1.x to 5.x.
17+
18+
==== Configuration file changes
19+
20+
Beats 5.0 come with a more powerful and more robust <<config-file-format,file
21+
configuration format>>. While this necessarily causes a few breaking changes, we
22+
provide a migration script that should make upgrading easy. Please see the
23+
<<upgrading>> section.
24+
25+
For consistency and clarity, all configuration options that are specific to a
26+
particular Beat are now prefixed by the name of the Beat. For example, the
27+
`interfaces.device` option from Packetbeat becomes
28+
`packetbeat.interfaces.device`. This change is made automatically by the
29+
migration script.
30+
31+
Similarly, common settings available in all Beats are now top level. As part of
32+
this, the options under the `shipper` section are moved to the top level. For
33+
example, the `shipper.name` settings becomes `name`. This change is made
34+
automatically by the migration script.
35+
36+
For consistency with the rest of the Elastic stack, several TLS/SSL related
37+
options where changed:
38+
39+
* the name of the section was renamed from `tls` to `ssl`
40+
* `certificate_key` was renamed to `key`
41+
* `insecure` was replaced by `verification_mode`
42+
* `min_version` and `max_version` were replaced with `supported_protocols`
43+
44+
These changes are made automatically by the migration script.
45+
46+
In Filebeat, we refactored the configuration around when to close and when to
47+
start ignoring files. Options like `close_older` and `force_close_older` were
48+
changed or replaced with options from the `close_*` group. See the
49+
{filebeat}/how-filebeat-works.html[How Filebeat works] docs for details. These
50+
changes are not made automatically by the migration script.
51+
52+
==== Field renames
53+
54+
Beats 5.0 also come with several changes in the exported fields. If you are
55+
using these fields in Kibana dashboards, you will likely need to adjust the
56+
dashbaords.
57+
58+
Changes:
59+
60+
[options="header",]
61+
|==========================================================
62+
| Location | Old name | New Name
63+
| Packetbeat HTTP module | http.phrase | http.response.phrase
64+
| Packetbeat HTTP module | http.code | http.response.code
65+
| Packetbeat HTTP module | http.response_headers | http.response.headers
66+
| Packetbeat HTTP module | http.request_headers | http.request.headers
67+
| Packetbeat HTTP module | http.params | http.request.params
68+
| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
69+
| Packetbeat HTTP module | http.content_length | http.response.headers.content-length
70+
| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
71+
| Packetbeat DNS module | dns.flags.recursion_allowed | dns.flags.recursion_available
72+
|==========================================================
73+
74+
In addition, the unused `count` field was removed from all Packetbeat modules.
75+
76+
==== Installation paths
77+
78+
For 5.0 we have refactored the {filebeat}/directory-layout.html[directory
79+
layout] of the Beats, so they are more consistent and in-line with the best
80+
practices of each operating system.
81+
82+
A notable change is that on the DEB/RPM installations, the binary is no longer
83+
installed in `/usr/bin` but in `/usr/share/<beatname>/bin`. Instead, a simple
84+
shell script is placed under `/usr/bin` that is helpful for executing the Beat
85+
in foreground. It also sets the path settings to the same values as the init
86+
script would use.
87+
88+
For example, here are the contents of the `/usr/bin/filebeat.sh` script:
89+
90+
91+
[source,shell]
92+
------------------------------------------------------------------------------
93+
> cat /usr/bin/filebeat.sh
94+
#!/bin/bash
95+
96+
# Script to run filebeat in foreground with the same path settings that
97+
# the init script / systemd unit file would do.
98+
99+
/usr/share/filebeat/bin/filebeat \
100+
-path.home /usr/share/filebeat \
101+
-path.config /etc/filebeat \
102+
-path.data /var/lib/filebeat \
103+
-path.logs /var/log/filebeat \
104+
$@
105+
------------------------------------------------------------------------------

libbeat/docs/index.asciidoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ include::./installing-beats.asciidoc[]
2323

2424
include::./repositories.asciidoc[]
2525

26+
include::./breaking.asciidoc[]
27+
28+
include::./upgrading.asciidoc[]
29+
2630
include::./config-file-format.asciidoc[]
2731

2832
// include::./visualizing-data.asciidoc[]

libbeat/docs/upgrading.asciidoc

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
[[upgrading]]
2+
== Upgrading
3+
4+
This section gives general recommendations for upgrading the Beats.
5+
6+
=== Upgrading between minor versions
7+
8+
As a general rule, upgrading between minor versions (e.g. 5.x to 5.y, where x <
9+
y) can be done by simply installing the new release and restarting the Beat
10+
process, because the Beats keep backwards compatibility when it comes to
11+
configuration and exported fields. Please review the <<release-notes,release
12+
notes>> for potential exceptions.
13+
14+
While we keep the backwards compatibility when it comes to the exported fields,
15+
a new minor version can add new fields (e.g. a new Metricbeat module is added).
16+
This can be problematic if the Elasticsearch mapping settings are not updated as
17+
well.
18+
19+
Please see the <<upgrade-mapping-template>> section for possible strategies in
20+
upgrading the Elasticsearch mapping templates.
21+
22+
23+
=== Upgrading from 1.x to 5.x
24+
25+
Before upgrading your Beats, please review the <<breaking-changes, breaking
26+
changes>> doc.
27+
28+
In addition to the subsections here, please read the
29+
<<upgrade-mapping-template>>
30+
31+
==== Configuration file migration
32+
33+
Beats 5.0 come with several backwards incompatible configuration changes.
34+
However, we provide a script that can automatically fixup the most common
35+
settings in your configuration file.
36+
37+
This script can be found in the Beats download packages. For the `.tar.gz` and
38+
`.zip` archives, it is found under:
39+
40+
[source,shell]
41+
------------------------------------------------------------------------------
42+
./scripts/migrate_beat_config_1_x_to_5_0.py
43+
------------------------------------------------------------------------------
44+
45+
For the `.deb` and `.rpm` packages it is found under the
46+
`/usr/share/<beatname>/scripts/` folder, for example:
47+
48+
49+
[source,shell]
50+
------------------------------------------------------------------------------
51+
/usr/share/packetbeat/scripts/migrate_beat_config_1_x_to_5_0.py
52+
------------------------------------------------------------------------------
53+
54+
Python 2.7 or later is required for executing the migration script.
55+
56+
To upgrade your configuration file automatically, simply pass it as an argument
57+
to the migration script, e.g.:
58+
59+
[source,shell]
60+
------------------------------------------------------------------------------
61+
> ./scripts/migrate_beat_config_1_x_to_5_0.py packetbeat.yml
62+
Backup file created: packetbeat.yml.bak
63+
------------------------------------------------------------------------------
64+
65+
The script will automatically create a backup file and overwrite it with the
66+
upgraded version. You can also use the `--dry` option, which doesn't modify the
67+
input file, but outputs the upgraded version to stdout instead.
68+
69+
==== Filebeat registry file location and format
70+
71+
Filebeat stores the read offset in a registry file that it persist to disk. The
72+
format of this file has changed in version 5.0, but Filebeat automatically
73+
upgrades the format on first run, so there's normally no action required on
74+
upgrading.
75+
76+
If you have installed Filebeat from the DEB or RPM packages, the location of the
77+
file stays the same as well (`/var/lib/filebeat/registry`). Also if you've
78+
installed Filebeat as a service on Windows the location stays the same
79+
(`C:\ProgramData\filebeat\registry`).
80+
81+
If you are running Filebeat from the `tar.gz` archive, however, the location of
82+
the registry file has changed. In version 1.x it was by default named
83+
`.filebeat` in the current working directory. In version 5.0, it is
84+
`data/registry` relative to the binary location. For upgrading in this case, we
85+
recommend copying the `.filebeat` file to `data/registry`.
86+
87+
==== Upgrading Topbeat to Metricbeat
88+
89+
With the Beats 5.0 release, Topbeat is replaced by Metricbeat, which offers more
90+
functionality. More precisely, the Topbeat functionality is roughly equivalent
91+
with the "system" module in Metricbeat.
92+
93+
While there is no automatic way to convert a Topbeat configuration file in the
94+
equivalent Metricbeat configuration file, both configuration are simple enough
95+
that migrating shouldn't take long.
96+
97+
For example, this is the input section of the Topbeat configuration file:
98+
99+
[source,yaml]
100+
------------------------------------------------------------------------------
101+
input:
102+
# In seconds, defines how often to read server statistics
103+
period: 10
104+
105+
# Regular expression to match the processes that are monitored
106+
# By default, all the processes are monitored
107+
procs: [".*"]
108+
109+
# Statistics to collect (all enabled by default)
110+
stats:
111+
# per system statistics, by default is true
112+
system: true
113+
114+
# per process statistics, by default is true
115+
process: true
116+
117+
# file system information, by default is true
118+
filesystem: true
119+
120+
# cpu usage per core, by default is false
121+
cpu_per_core: false
122+
------------------------------------------------------------------------------
123+
124+
And here is the equivalent configuration for the system module in Metricbeat:
125+
126+
[source,yaml]
127+
------------------------------------------------------------------------------
128+
metricbeat.modules:
129+
130+
#------------------------------- System Module -------------------------------
131+
- module: system
132+
metricsets:
133+
# CPU stats
134+
- cpu
135+
136+
# System Load stats
137+
- load
138+
139+
# Per filesystem stats
140+
- filesystem
141+
142+
# Memory stats
143+
- memory
144+
145+
# Network stats
146+
- network
147+
148+
# Per process stats
149+
- process
150+
enabled: true
151+
period: 10s
152+
processes: ['.*']
153+
------------------------------------------------------------------------------
154+
155+
The rest of the configuration options (outputs, name, tags) stay the same or can
156+
be upgraded using the migration script.
157+
158+
[[upgrade-mapping-template]]
159+
=== Upgrade the Elasticsearch mapping template
160+
161+
By default, the Elasticsearch mapping template which comes with the Beats is not
162+
re-installed if it already exists. This means that if the new Beats version
163+
adds new fields (e.g. a new module in Metricbeat), they won't be reflected in
164+
the Elasticsearch mapping, which can result in visualization problems or
165+
incorect data.
166+
167+
You can set the `output.elasticsearch.template.overwrite` option to `true` to
168+
make the Beats install the new mapping template even if a template with the same
169+
name already exists. You can set this option in the configuration file or as a
170+
CLI flag, for example:
171+
172+
[source,shell]
173+
------------------------------------------------------------------------------
174+
packetbeat -e -E output.elasticsearch.template.overwrite=true
175+
------------------------------------------------------------------------------
176+
177+
However, the mapping template is applied on index creation, so it won't affect
178+
the current index if it was already created with the old version.
179+
180+
You can force the start of a new index by renaming the old index before starting
181+
the new version of the Beat. As of Elasticsearch 5.0, this can be done via the
182+
re-index API:
183+
184+
185+
[source,json]
186+
------------------------------------------------------------------------------
187+
POST /_reindex
188+
{
189+
"source": {
190+
"index": "packetbeat-2016.09.20"
191+
},
192+
"dest": {
193+
"index": "packetbeat-2016.09.20-old"
194+
}
195+
}
196+
DELETE /packetbeat-2016.09.20
197+
------------------------------------------------------------------------------
198+
199+
Note that the reindex API command can take a long time, depending on the size of
200+
the index. It is recommended that the Beat is stopped during this time, so the
201+
order of operations should be:
202+
203+
1. Stop the old version of the Beat
204+
2. Rename the index of the current day
205+
3. Start the new version of the Beat
206+
207+
If downtime is not acceptable, another possible approach is to configure a
208+
different index pattern in the new Beat version, but this will likely require
209+
adjustments to your Kibana dashboards.

0 commit comments

Comments
 (0)