encoding/xml: add whitespace normalization from spec

[santhosh-tekuri]

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64

What operating system and processor architecture are you using (go env)?

darwin/amd64

What did you do?

for attribute values, all whitespace characters (#x20, #xD, #xA, #x9), should be normalized to a space character (see https://www.w3.org/TR/REC-xml/#AVNormalize)

go program used to test: (https://play.golang.org/p/mCRrxvyh25)

package main

import (
	"encoding/xml"
	"fmt"
	"strings"
)

func main() {
	decoder := xml.NewDecoder(strings.NewReader(`<a p1="v1

v1"/>`))
	t, _ := decoder.Token()
	if e, ok := t.(xml.StartElement); ok {
		fmt.Printf("%q", e.Attr[0].Value)
	}
}

What did you expect to see?

"v1  v1"

What did you see instead?

"v1\n\nv1"

Tested the same with xmllint program:

$ cat test.xml
<x p1="v1

v2"/>
$ xmllint test.xml
<?xml version="1.0"?>
<x p1="v1  v2"/>

[bradfitz]

/cc @rsc for XML.

[rsc]

The spec seems very clear that this is required, and we're not doing it. Seems OK to do. Note the oddity that

<x a="
v">
<x a="&#x0Av">

are not equivalent. The literal newline turns into a space but the escaped newline remains a newline.

It also looks like we are not handling end-of-line \r correctly (https://www.w3.org/TR/REC-xml/#sec-line-ends).

CLs welcome for Go 1.10 (but please keep it simple).

[Kinghack]

could I try this?

[bradfitz]

Go for it.

[Kinghack]

I am trying to verify my diff.
I found that with xmllint, they are not dealing with leading and trailing space (#x20).
Example is

<x p1=" v1

\nv2

" p2=" v2 v1 "/>

The result with
xmllint test2.xml
is
<?xml version="1.0"?> <x p1=" v1 \nv2 " p2=" v2 v1 "/>

Do I misunderstand the spec?

[rsc]

The spec says:

If the attribute type is not CDATA, then the XML processor must further process the normalized attribute value by discarding any leading and trailing space (#x20) characters, and by replacing sequences of space (#x20) characters by a single space (#x20) character.

But for our purposes the attribute type is always CDATA, since we don't parse the related entity declarations. So that sentence does not apply.

[gopherbot]

CL https://golang.org/cl/46433 mentions this issue.

Hi I've been looking at this issue and the patch that has been submitted ( https://go-review.googlesource.com/c/46433/ ) and there is a problem. The patch is modifying the text function of xml.go, this function is called both for character data nodes and for attribute values. The problem is that the normalization for character data and attribute values is different and the patch is applying the attribute value normalization to character data.

This is an error, since for character data the spec states (https://www.w3.org/TR/REC-xml/#sec-white-space):

"In editing XML documents, it is often convenient to use "white space" (spaces, tabs, and blank lines) to set apart the markup for greater readability. Such white space is typically not intended for inclusion in the delivered version of the document. On the other hand, "significant" white space that should be preserved in the delivered version is common, for example in poetry and source code.

An XML processor MUST always pass all characters in a document that are not markup through to the application..."

The only normalization that must be done on Character Data nodes is the end of line handling:

https://www.w3.org/TR/REC-xml/#sec-line-ends

On the other hand for attribute values both the end of line handling and the attribute value normalization (https://www.w3.org/TR/REC-xml/#AVNormalize) must be done.

But applying the attribute values normalization to text nodes is incorrect and will create all kinds of problems on client applications. This is evident by the changes that the patch does to tests, for example the atom feed test here: https://go-review.googlesource.com/c/46433/4/src/encoding/xml/read_test.go#b119

Anyhow, I have read the spec and the code extensively, I think I know how to fix both issues (end of line on both character data and attribute values and attribute value normalization on attribute values) simply and fast, so if this is still open I would love to give it a try.