|
| 1 | +[[upgrading]] |
| 2 | +== Upgrading |
| 3 | + |
| 4 | +This section gives general recommendations for upgrading the Beats. |
| 5 | + |
| 6 | +=== Upgrading between minor versions |
| 7 | + |
| 8 | +As a general rule, upgrading between minor versions (e.g. 5.x to 5.y, where x < |
| 9 | +y) can be done by simply installing the new release and restarting the Beat |
| 10 | +process, because the Beats keep backwards compatibility when it comes to |
| 11 | +configuration and exported fields. Please review the <<release-notes,release |
| 12 | +notes>> for potential exceptions. |
| 13 | + |
| 14 | +While we keep the backwards compatibility when it comes to the exported fields, |
| 15 | +a new minor version can add new fields (e.g. a new Metricbeat module is added). |
| 16 | +This can be problematic if the Elasticsearch mapping settings are not updated as |
| 17 | +well. |
| 18 | + |
| 19 | +Please see the <<upgrade-mapping-template>> section for possible strategies in |
| 20 | +upgrading the Elasticsearch mapping templates. |
| 21 | + |
| 22 | + |
| 23 | +=== Upgrading from 1.x to 5.x |
| 24 | + |
| 25 | +Before upgrading your Beats, please review the <<breaking-changes, breaking |
| 26 | +changes>> doc. |
| 27 | + |
| 28 | +In addition to the subsections here, please read the |
| 29 | +<<upgrade-mapping-template>> |
| 30 | + |
| 31 | +==== Configuration file migration |
| 32 | + |
| 33 | +Beats 5.0 come with several backwards incompatible configuration changes. |
| 34 | +However, we provide a script that can automatically fixup the most common |
| 35 | +settings in your configuration file. |
| 36 | + |
| 37 | +This script can be found in the Beats download packages. For the `.tar.gz` and |
| 38 | +`.zip` archives, it is found under: |
| 39 | + |
| 40 | +[source,shell] |
| 41 | +------------------------------------------------------------------------------ |
| 42 | +./scripts/migrate_beat_config_1_x_to_5_0.py |
| 43 | +------------------------------------------------------------------------------ |
| 44 | + |
| 45 | +For the `.deb` and `.rpm` packages it is found under the |
| 46 | +`/usr/share/<beatname>/scripts/` folder, for example: |
| 47 | + |
| 48 | + |
| 49 | +[source,shell] |
| 50 | +------------------------------------------------------------------------------ |
| 51 | +/usr/share/packetbeat/scripts/migrate_beat_config_1_x_to_5_0.py |
| 52 | +------------------------------------------------------------------------------ |
| 53 | + |
| 54 | +Python 2.7 or later is required for executing the migration script. |
| 55 | + |
| 56 | +To upgrade your configuration file automatically, simply pass it as an argument |
| 57 | +to the migration script, e.g.: |
| 58 | + |
| 59 | +[source,shell] |
| 60 | +------------------------------------------------------------------------------ |
| 61 | +> ./scripts/migrate_beat_config_1_x_to_5_0.py packetbeat.yml |
| 62 | +Backup file created: packetbeat.yml.bak |
| 63 | +------------------------------------------------------------------------------ |
| 64 | + |
| 65 | +The script will automatically create a backup file and overwrite it with the |
| 66 | +upgraded version. You can also use the `--dry` option, which doesn't modify the |
| 67 | +input file, but outputs the upgraded version to stdout instead. |
| 68 | + |
| 69 | +==== Filebeat registry file location and format |
| 70 | + |
| 71 | +Filebeat stores the read offset in a registry file that it persist to disk. The |
| 72 | +format of this file has changed in version 5.0, but Filebeat automatically |
| 73 | +upgrades the format on first run, so there's normally no action required on |
| 74 | +upgrading. |
| 75 | + |
| 76 | +If you have installed Filebeat from the DEB or RPM packages, the location of the |
| 77 | +file stays the same as well (`/var/lib/filebeat/registry`). Also if you've |
| 78 | +installed Filebeat as a service on Windows the location stays the same |
| 79 | +(`C:\ProgramData\filebeat\registry`). |
| 80 | + |
| 81 | +If you are running Filebeat from the `tar.gz` archive, however, the location of |
| 82 | +the registry file has changed. In version 1.x it was by default named |
| 83 | +`.filebeat` in the current working directory. In version 5.0, it is |
| 84 | +`data/registry` relative to the binary location. For upgrading in this case, we |
| 85 | +recommend copying the `.filebeat` file to `data/registry`. |
| 86 | + |
| 87 | +==== Upgrading Topbeat to Metricbeat |
| 88 | + |
| 89 | +With the Beats 5.0 release, Topbeat is replaced by Metricbeat, which offers more |
| 90 | +functionality. More precisely, the Topbeat functionality is roughly equivalent |
| 91 | +with the "system" module in Metricbeat. |
| 92 | + |
| 93 | +While there is no automatic way to convert a Topbeat configuration file in the |
| 94 | +equivalent Metricbeat configuration file, both configuration are simple enough |
| 95 | +that migrating shouldn't take long. |
| 96 | + |
| 97 | +For example, this is the input section of the Topbeat configuration file: |
| 98 | + |
| 99 | +[source,yaml] |
| 100 | +------------------------------------------------------------------------------ |
| 101 | +input: |
| 102 | + # In seconds, defines how often to read server statistics |
| 103 | + period: 10 |
| 104 | +
|
| 105 | + # Regular expression to match the processes that are monitored |
| 106 | + # By default, all the processes are monitored |
| 107 | + procs: [".*"] |
| 108 | +
|
| 109 | + # Statistics to collect (all enabled by default) |
| 110 | + stats: |
| 111 | + # per system statistics, by default is true |
| 112 | + system: true |
| 113 | +
|
| 114 | + # per process statistics, by default is true |
| 115 | + process: true |
| 116 | +
|
| 117 | + # file system information, by default is true |
| 118 | + filesystem: true |
| 119 | +
|
| 120 | + # cpu usage per core, by default is false |
| 121 | + cpu_per_core: false |
| 122 | +------------------------------------------------------------------------------ |
| 123 | + |
| 124 | +And here is the equivalent configuration for the system module in Metricbeat: |
| 125 | + |
| 126 | +[source,yaml] |
| 127 | +------------------------------------------------------------------------------ |
| 128 | +metricbeat.modules: |
| 129 | +
|
| 130 | +#------------------------------- System Module ------------------------------- |
| 131 | +- module: system |
| 132 | + metricsets: |
| 133 | + # CPU stats |
| 134 | + - cpu |
| 135 | +
|
| 136 | + # System Load stats |
| 137 | + - load |
| 138 | +
|
| 139 | + # Per filesystem stats |
| 140 | + - filesystem |
| 141 | +
|
| 142 | + # Memory stats |
| 143 | + - memory |
| 144 | +
|
| 145 | + # Network stats |
| 146 | + - network |
| 147 | +
|
| 148 | + # Per process stats |
| 149 | + - process |
| 150 | + enabled: true |
| 151 | + period: 10s |
| 152 | + processes: ['.*'] |
| 153 | +------------------------------------------------------------------------------ |
| 154 | + |
| 155 | +The rest of the configuration options (outputs, name, tags) stay the same or can |
| 156 | +be upgraded using the migration script. |
| 157 | + |
| 158 | +[[upgrade-mapping-template]] |
| 159 | +=== Upgrade the Elasticsearch mapping template |
| 160 | + |
| 161 | +By default, the Elasticsearch mapping template which comes with the Beats is not |
| 162 | +re-installed if it already exists. This means that if the new Beats version |
| 163 | +adds new fields (e.g. a new module in Metricbeat), they won't be reflected in |
| 164 | +the Elasticsearch mapping, which can result in visualization problems or |
| 165 | +incorect data. |
| 166 | + |
| 167 | +You can set the `output.elasticsearch.template.overwrite` option to `true` to |
| 168 | +make the Beats install the new mapping template even if a template with the same |
| 169 | +name already exists. You can set this option in the configuration file or as a |
| 170 | +CLI flag, for example: |
| 171 | + |
| 172 | +[source,shell] |
| 173 | +------------------------------------------------------------------------------ |
| 174 | +packetbeat -e -E output.elasticsearch.template.overwrite=true |
| 175 | +------------------------------------------------------------------------------ |
| 176 | + |
| 177 | +However, the mapping template is applied on index creation, so it won't affect |
| 178 | +the current index if it was already created with the old version. |
| 179 | + |
| 180 | +You can force the start of a new index by renaming the old index before starting |
| 181 | +the new version of the Beat. As of Elasticsearch 5.0, this can be done via the |
| 182 | +re-index API: |
| 183 | + |
| 184 | + |
| 185 | +[source,json] |
| 186 | +------------------------------------------------------------------------------ |
| 187 | +POST /_reindex |
| 188 | +{ |
| 189 | + "source": { |
| 190 | + "index": "packetbeat-2016.09.20" |
| 191 | + }, |
| 192 | + "dest": { |
| 193 | + "index": "packetbeat-2016.09.20-old" |
| 194 | + } |
| 195 | +} |
| 196 | +DELETE /packetbeat-2016.09.20 |
| 197 | +------------------------------------------------------------------------------ |
| 198 | + |
| 199 | +Note that the reindex API command can take a long time, depending on the size of |
| 200 | +the index. It is recommended that the Beat is stopped during this time, so the |
| 201 | +order of operations should be: |
| 202 | + |
| 203 | +1. Stop the old version of the Beat |
| 204 | +2. Rename the index of the current day |
| 205 | +3. Start the new version of the Beat |
| 206 | + |
| 207 | +If downtime is not acceptable, another possible approach is to configure a |
| 208 | +different index pattern in the new Beat version, but this will likely require |
| 209 | +adjustments to your Kibana dashboards. |
0 commit comments