Major Changes for Downstream

[alexr00]

It looks like new used to be keyword.control.cpp. Now it is keyword.operator.memory.new.cpp. Is this intentional? If it's intentional, we can add the it into the themes in VS Code, but ideally there wouldn't be too many changes like this since it requires constant updating of the themes.

[alexr00]

Original issue: microsoft/vscode#71821

[jeff-hykin]

Yes it was intentional, and I agree we will try to keep changes like this to a minimum since many themes already setup with the old way.

From here on out, I'll put any similar changes on this post.

C++ Log as of August 12th 2019

• Helper-tags have been added to make tag changes much easier to maintain. None of these change/replace existing tags
  • (1.7.0) keyword.operator.functionlike.cpp keyword.operator
    will match all of the operators like alignas, sizeof, etc that require parentheses
  • (1.7.0) keyword.operator.wordlike.cpp keyword.operator
    will match all of the operators that use alpha numeric characters (that are not functionlike)
  • (1.7.0) memory keyword.operator
    will match both keyword.operator.new and keyword.operator.delete
• new and delete changed
  • (1.0.0) keyword.control
  • (1.4.0) keyword.operator.memory.new
  • (1.7.0) keyword.operator.new and keyword.operator.delete
• The using keyword changed
  • (1.0.0) keyword.control
  • (1.4.0) keyword.other.using
• and, or, xor, etc have received additional specificity
  • (1.7.0) match them with alias keyword.operator
  • (1.11.7) match them with keyword.operator.wordlike keyword.operator or keyword.operator.or
• The operator keyword was changed
  • (1.0.0) keyword.control
  • (1.5.0) keyword.other.operator
• Member access received additional scopes
  • (1.4.8) variable.other.object.accessor
    • To effectively revert this set the color of this tag to be the same as source.cpp
• Member variables have changed
  • (1.0.0) variable.other.member
  • (1.4.8) variable.other.object.property (middle) and variable.other.property (end)
    • this was done to better accommodate Local variables are randomly highlighted incorrectly. #59 and be consistent across languages (see Java, Javascript, C#, Typescript, CoffeeScript)
• The namespace {}'s have changed
  • (1.0.0) punctuation.definition.scope
  • (1.7.0) punctuation.section.block.begin.bracket.curly.namespace.cpp punctuation.section.block.end.bracket.curly.namespace.cpp
    • This change is because class/struct/enum/union/extern all use that for their brackets. Namespace brackets are the only anti-pattern
• decltype is now tagged
  • (1.6.0) both storage.type.decltype and keyword.other.decltype
• The class, struct, union, enum, extern, namespace pattern(s) were overhauled and standardized
  • (1.0.0) meta.class-struct-block, meta.extern-block, meta.namespace-block
  • (1.7.0) changed to
    • meta.block.class
    • meta.block.struct
    • meta.block.union
    • meta.block.enum
    • meta.block.extern
    • meta.block.namespace
• scope of namespaces has changed
  • (1.8.2) entity.name.type.namespace
  • (1.9.1) entity.name.namespace
• pthread types have changed
  • (1.0) support.type.pthread
  • (1.8.5) support.type.posix-reserved.pthread
• change to ()'s
  • (1.0) punctuation.section.parens
  • (1.8.6) meta.parens.c
  • [ subject to change in future, it will stay meta, but might be meta.block or similar ]
• function pointers
  • (1.0) no tagging
  • (1.8.14) experimental tagging inside of structs/classes only
  • (1.9.0) tagging inside of function parameters
• pointer and reference
  • (1.0) keyword.operator
  • (1.8.14) storage.modifier.pointer and storage.modifier.reference
• scope resolution identifiers
  • (1.0?) entity.name.type.namespace.scope-resolution
  • (1.9.0)
    • entity.name.scope-resolution
    • entity.name.scope-resolution.function.call
    • entity.name.scope-resolution.function.definition
    • entity.name.scope-resolution.function.definition.operator-overload
    • entity.name.scope-resolution.namespace.alias
    • entity.name.scope-resolution.namespace.block
    • entity.name.scope-resolution.namespace.using
    • entity.name.scope-resolution.template.call
    • entity.name.scope-resolution.template.definition
• scope resolution punctuation ::
  • (1.0) punctuation.separator.namespace.access
  • (1.9.0)
    • punctuation.separator.scope-resolution
    • punctuation.separator.scope-resolution.function.call
    • punctuation.separator.scope-resolution.function.definition
    • punctuation.separator.scope-resolution.function.definition.operator-overload
    • punctuation.separator.scope-resolution.namespace.alias
    • punctuation.separator.scope-resolution.namespace.block
    • punctuation.separator.scope-resolution.namespace.using
    • punctuation.separator.scope-resolution.template.call
    • punctuation.separator.scope-resolution.template.definition
  • (1.9.2) planned to add punctuation.separator.namespace.access for better backwards compatibility
• template call angle brackets
  • (1.0) keyword.operator.comparision
  • (1.9.0) more of them are now tagged as
    • punctuation.section.angle-brackets.begin.template.call
    • punctuation.section.angle-brackets.end.template.call
    • punctuation.section.angle-brackets.begin.template.definition
    • punctuation.section.angle-brackets.end.template.definition
• macro parentheses
  • (1.0) punctuation.section.arguments.begin.bracket.round.cpp
  • (1.0) punctuation.section.arguments.end.bracket.round.cpp
  • (1.13.0) punctuation.definition.parameters.begin.preprocessor
  • (1.13.0) punctuation.definition.parameters.end.preprocessor

[jeff-hykin]

The reason it was changed is because new and delete are completely unrelated to control flow. I picked operator since that's what what I believed it was classified under the C++ spec.

@matter123 pointed out in his pull request its only an operator under a special case. Additionally its debatable that the coloration would be improved if it were labeled with keyword.other.

@alexr00 should we go ahead and make the change from keyword.operator to keyword.other?

[alexr00]

Having the big changes here is perfect, thank you. I took a quick look at how other languages handle new:
coffee
keyword.operator.new.coffee

JS
keyword.operator.new.js

groovy
keyword.control.new.groovy

java
keyword.control.new.java

There isn't a real consensus. Based on what the VS Code themes already have, keyword.operator.new would be good. keyword.other, like keyword.operator.memory would still require changes to the theme. I'm happy to make minor updates to themes though. If you prefer keyword.operator.memory more, then feel free to leave it and I'll make the update to include it here and in the equivalent places in the other themes.

[matter123]

keyword.operator.new is probably the best. I noticed on [expr.new]/4 the standard refers to the new operator. Though change is still needed for keyword.operator.delete

[jeff-hykin]

Thats interesting @alexr00 I'll try to get a complete list of changes by the end of tonight. There is a file in this repo syntax_examples.cpp it is designed to visually debug the syntax highlighter changes. You can probably use it to quickly see how everything looks.

[jeff-hykin]

@alexr00 The list isn't 100% complete, but I believe it covers everything that would be considered an issue. Many of the changes were made tonight to try to get all of the changes done now. I probably won't be able to make changes for about a week or more, but I'll be around daily for GitHub communication.

Do note that these changes are C++ exclusive. We are working on an ideal way to port them to C.

I believe the new keyword.operator.wordlike, keyword.operator.functionlike should take care of most of the issues.

[jeff-hykin]

Today I added a tags.txt, that gets auto generated with the syntax. It just sorts and lists out all of the tags that are in the grammar, it should be useful for keeping track of when things are added and removed. @matter123