Ensuring we are up to date with x-common

[petertseng]

How should we ensure we're up to date?

One idea is: Every time someone makes a change in x-common, that person is expected to file an issue against all tracks that implement the exercise. If we rely on this to happen, then we need take no further action.

Sometimes, that is not always adhered to. We mark dates in our test files, what if we used that to an advantage?

I wrote a short script in exercism/problem-specifications@master...petertseng:up-to-date . Sorry it is not in Haskell, but the idea should be clear:

• read out the date from our test file
• ask x-common if that canonical-data.json file has been changed since that date

In the short term: I will be making some PRs today based on what I've found.
In the long term: Which route shall we take?

[rbasso]

The best solution I could find until now is to add the data to config.json:

  "exercises": [
    {
      "slug": "leap",
      "difficulty": 1,
      "topics": [
      ],
      "x-common-file": "exercises/leap/canonical-data.json",
      "x-common-hash": "cda8f9800a33d997f8c6146a10b8caf66e25ec4b"
    },
    ...

With this information there, it would be easier to match against x-common:

$ git diff cda8f9800a33d997f8c6146a10b8caf66e25ec4b exercises/leap/canonical-data.json

The only thing we would have to keep in the test suite are the deviations from x-common.

What do you think about it?

[rbasso]

As an example, this shamefull bash script gets the filenames and hashes from config.json and checks which files changed in the repository:

Beware: Ugly script ahead!

checkChanges

#!/usr/bin/env bash


QUERY='.exercises|map(select(has("x-common-file")))|map({"x-common-hash","x-common-file"})|map("git diff --name-only " + ."x-common-hash" + " " + ."x-common-file" + ";") | .[]';

eval $(jq -r "$QUERY" "$@")

It must be run from the root of x-common, passing the config.json file as an argument:

.../x-common/ $
.../x-common/ $ checkChanges ../xhaskell/config.json
exercises/leap/canonical-data.json
exercises/space-age/canonical-data.json
...

[petertseng]

It seems good to use a sha instead of a date - it allows us to check precisely whether the file has changed. That is good.

It may not be necessary to specify the file directly, since they are all "exercises/" ++ slug ++ "/canonical-data.json". I would say it only needs to be specified if it breaks the pattern.

If we move to using the config.json file, I assume we would delete the date lines from the individual Tests.hs files? I wouldn't want to have to maintain our data in two places.

[rbasso]

It may not be necessary to specify the file directly, since they are all "exercises/" ++ slug ++ "/canonical-data.json". I would say it only needs to be specified if it breaks the pattern.

In case of a change in the directory structure we'll have to think about how to deal with that. Considering that most of the json where not changed after the last restructuring, maybe we need to test if it will work without the file.

If we move to using the config.json file, I assume we would delete the date lines from the individual Tests.hs files?

Agreed!

[petertseng]

In case of a change in the directory structure we'll have to think about how to deal with that.

I am assuming that if there is a change that all the files change at once (like what happened with canonical-data.json). In this case, all we need to do is change the file our script looks at, I think?

I will say that in my current script I make use of git log --follow to follow renames, but since we want to use diff to see whether the file changed, I guess we can't do that.

I admit that I'm not thinking very hard about this because I expect a change in directory structure to be a rare event, and the benefit is that we avoid having to explicitly specify the JSON file. I could be convinced about explicitly specifying if you could convince me about how it deals with structure changes

[rbasso]

Ok...I think we can take the risk and forget about the filenames for now. 😄

[petertseng]

OK, so I guess if we want to move to this scheme we just need to start by removing all the date lines and filling in commit SHAs. Should we:

1. Just use the current HEAD of x-common? The only exercises for which we are out of date currently are word-count and triangle, I think. So we would just use different commits for those two exercises.
2. Look at the current date of each exercise and try to find in a commit near that date? Seems more work.

[rbasso]

The current SHA seems good enough.

I don't care too much about what is currently not following x-common. Detecting changes from now on seems good enough.

[petertseng]

Going to ask real quick, thoughts on placing all the data in config.json instead of the individual test files?

config.json:

• won't get shown to students, so they won't see irrelevant info they don't need to worry about
• is all in one file, if that helps (does it?)
• very easy to understand how to parse it (JSON), not clear how to find it if we bury it in the file (just grep for 40 [0-9a-f] characters in a row?)

individual test files:

• probably harder to forget to change it
• if I ask the question "when was accumulate updated?" it's probably easier to understand if I know I look in accumuate's tests, instead of look for it in JSON

It's probably not that hard to write something to insert in the x-common-hash to all exercises right now.
Note it's not a super high priority for me right now since currently looking at the date works and I honestly think that we (mostly you) made sure we were up to date pretty well when doing the hspec rewrites. So I might focus on some other things first.

[rbasso]

Humm....I think that the deviations from x-common should stay in the test suite. It's not a problem if the students look at it, in fact it could be useful, because a lot of them already know the exercise from other tracks, also they can become contributors.

Yeah...It is probably better put put everything in the test suite.
I'm not sure about the format and position.

[petertseng]

I should probably mention this here while we are thinking about this subject - another alternative is to write test generators for those exercises that have canonical-data.json

I have to honestly say that my current perceived benefit of writing the test generator is not high enough to justify the costs that I anticipate. I'm willing to have my mind changed on that, of course.

[rbasso]

The frequency of updates to exercises in this track is high, but most of the changes are not caused by minor changes in x-common. I think that, if we automatize test generation, we would innovate less.