Update geoip databases to include data about documentation ranges #2450

mrodm · 2025-03-04T11:40:10Z

Closes #2414
Follows #2408

This PR includes new GeoIP databases including some entries for the documentation prefixes.

Added entries for the following address blocks for documentation in the GeoLite2 JSON files:

RFC
- 192.0.2.0/24
- 198.51.100.0/24
- 203.0.113.0/24
RFC3849
- 2001:DB8::/32
ASN for documentation defined in RFC

I've tried to use other "real" cities here (Las Vegas, Amsterdam and Madrid) as well as Greenwich (as mentioned in this blog post), and get the data about them using https://www.geonames.org/

This data set in those JSON files is faked just for testing purposes in elastic-package. This could affect system and pipeline tests.

Data added to the databases can be reviewed in this commit: 906b3a6

Moreover, some tests have been added to the isAllowedIPValue function as well as it has been used an IP of the documentation range in the test package apache.

Author's checklist

Add documentation
Add documentation prefixes into GeoIP databases
- This data can be reviewed at: 906b3a6

How to test this PR locally

# ensure there is no stack running before doing "stack up"
elastic-package stack down -v

elastic-package stack up -v -d

# test package "apache" includes now an IP from one of the documentation ranges.
cd test/packages/parallel/apache
elastic-package test -v

elastic-package stack down -v

mrodm · 2025-03-05T09:52:45Z

test integrations

elastic-vault-github-plugin-prod · 2025-03-05T10:01:49Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#12962

mrodm · 2025-03-05T17:08:40Z

internal/stack/_static/geoip_source/GeoLite2-City-Test.json

I've tried to use "real" cities here in the new entries. And for that, I got the data about them (geo codes or latitute/longitude values) using https://www.geonames.org/

The assignation of this ranges to those cities is faked and it is just for testing purposes in elastic-package.

Do you see any issue of using that data here?

Here it can be checked the specific data added (Las Vegas, Amsterdam and Madrid) and the Greenwich entry from this blog post:
906b3a6

I think it is fine to use real cities.

mrodm · 2025-03-05T17:21:46Z

test integrations

elastic-vault-github-plugin-prod · 2025-03-05T17:28:03Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#12962

mrodm · 2025-03-06T09:27:31Z

test integrations

elastic-vault-github-plugin-prod · 2025-03-06T09:32:57Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#12983

Include documentation about these GeoLite2 databases and how they should be generated.

jsoriano · 2025-03-07T09:37:08Z

internal/stack/_static/geoip_source/GeoLite2-City-Test.json

I think it is fine to use real cities.

jsoriano · 2025-03-07T09:58:53Z

internal/stack/_static/geoip_source/README.md

+
+Once applied all these changes, you can build the `write-test-data` tool:
+```shell
+git clone https://github.com/maxmind/MaxMind-DB


At this point the repository should have been already cloned, and patched.

With the latest changes, this section has been updated and now it just refers to the new tool.

internal/stack/_static/geoip_source/README.md

internal/stack/_static/geoip_source/GeoLite2-City-Test.json

mrodm · 2025-03-07T14:59:07Z

tools/geoipdatabases/writer.go

In https://github.com/maxmind/MaxMind-DB, this file and geoip2.go are located in another module (pkg/writer).

I thought to keep it simple here and just locate everything in just one package main.

mrodm · 2025-03-07T15:00:27Z

test/packages/parallel/apache/data_stream/access/_dev/test/pipeline/test-access-darwin.log

@@ -4,3 +4,4 @@
 89.160.20.156 - - [26/Dec/2016:18:23:35 +0200] "GET / HTTP/1.1" 200 45
 89.160.20.156 - - [26/Dec/2016:18:23:41 +0200] "GET /notfound HTTP/1.1" 404 206
 89.160.20.156 - - [26/Dec/2016:18:23:45 +0200] "GET /hmm HTTP/1.1" 404 201
+192.0.2.100 - - [26/Dec/2016:18:23:45 +0200] "GET /hmm HTTP/1.1" 404 201


New ip from the address block of documentation (192.0.2.100/24)

jsoriano

👍

jsoriano · 2025-03-07T16:18:41Z

tools/geoipdatabases/main.go

+	source := flag.String("source", "", "Source data directory")
+	target := flag.String("target", "", "Destination directory for the generated mmdb files")


Nit. We could hard-code these paths to the paths we have.

Done in f6799e4
and updated README accordingly.

elasticmachine · 2025-03-10T10:31:34Z

💚 Build Succeeded

Buildkite Build
Commit: f6799e4

History

💚 Build #4976 succeeded d8dd4d3
💚 Build #4975 succeeded 3332c0d
💔 Build #4974 failed f399f57
💔 Build #4973 failed ff56b13
💚 Build #4965 succeeded 8ae8411
💚 Build #4964 succeeded 17add30

cc @mrodm

mrodm added 2 commits March 4, 2025 10:50

Add tests for isAllowedIPValue

0056f06

Update test package apache to include IP in the documentation range

c8a277d

mrodm self-assigned this Mar 4, 2025

elastic-vault-github-plugin-prod bot mentioned this pull request Mar 5, 2025

Test elastic-package#2450 - DO NOT MERGE elastic/integrations#12962

Closed

mrodm commented Mar 5, 2025

View reviewed changes

mrodm mentioned this pull request Mar 5, 2025

Replace Geo IP database used in testing with dummy data #2414

Closed

elastic-vault-github-plugin-prod bot mentioned this pull request Mar 6, 2025

Test elastic-package#2450 - DO NOT MERGE elastic/integrations#12983

Closed

mrodm added 2 commits March 6, 2025 13:59

Add source JSON files with no modifications

e51c2f8

Update GeoIP databases including documentation prefixes

906b3a6

Include documentation about these GeoLite2 databases and how they should be generated.

mrodm force-pushed the update_geoip_databases branch from 88211de to 906b3a6 Compare March 6, 2025 13:18

Update README

17add30

mrodm changed the title ~~[system tests] Update geoip databases to include data about documentation ranges - WIP~~ [system tests] Update geoip databases to include data about documentation ranges Mar 6, 2025

Update README - Keep consistency in naming

8ae8411

mrodm changed the title ~~[system tests] Update geoip databases to include data about documentation ranges~~ Update geoip databases to include data about documentation ranges Mar 6, 2025

mrodm marked this pull request as ready for review March 6, 2025 16:56

mrodm requested a review from a team March 6, 2025 16:56

jsoriano reviewed Mar 7, 2025

View reviewed changes

mrodm added 3 commits March 7, 2025 12:47

Create tool to create/update geoipdatabased based on MaxMind-DB repo

ff56b13

Merge upstream/main into update_geoip_databases

f399f57

Remove unused variable

3332c0d

mrodm commented Mar 7, 2025

View reviewed changes

Add note

d8dd4d3

mrodm requested a review from jsoriano March 7, 2025 16:07

jsoriano approved these changes Mar 7, 2025

View reviewed changes

Hardcode elastic-package paths for databases and sources

f6799e4

mrodm merged commit 3d15e20 into elastic:main Mar 10, 2025
3 checks passed

mrodm deleted the update_geoip_databases branch March 10, 2025 14:47

mrodm mentioned this pull request Mar 14, 2025

Update GeoIP databases with more documentation ranges #2476

Merged

2 tasks

		source := flag.String("source", "", "Source data directory")
		target := flag.String("target", "", "Destination directory for the generated mmdb files")

Update geoip databases to include data about documentation ranges #2450

Update geoip databases to include data about documentation ranges #2450

Uh oh!

Conversation

mrodm commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

How to test this PR locally

Uh oh!

mrodm commented Mar 5, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Mar 5, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 5, 2025

Uh oh!

mrodm commented Mar 6, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mrodm Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Uh oh!

mrodm commented Mar 4, 2025 •

edited

Loading

mrodm Mar 7, 2025 •

edited

Loading