Package

eu.cdevreeze

yaidom

Permalink

package yaidom

Yaidom is yet another Scala immutable DOM-like XML API. Some other well-known Scala immutable DOM-like APIs are the standard scala.xml API and Anti-XML. The latter API is considered by many to be an improvement over the former, but both APIs:

Yaidom takes a different approach, avoiding XPath-like query support, and offering good namespace and decent (functional) update support. Yaidom is also characterized by almost mathematical precision and clarity. Still, the API remains practical and pragmatic. In particular, the API user has much configuration control over parsing and serialization, because yaidom exposes the underlying JAXP parsers and serializers, which can be configured by the library user.

Yaidom chooses its battles. For example, given that DTDs do not know about namespaces, yaidom offers good namespace support, but ignores DTDs entirely. Of course the underlying XML parser may still validate XML against a DTD, if so desired. As another example, yaidom tries to leave the handling of the gory details of XML processing (such as whitespace handling) as much as possible to JAXP (and JAXP parser/serializer configuration). As yet another example, yaidom knows nothing about (XML Schema) types of elements and attributes.

As mentioned above, yaidom tries to treat basic XML processing with almost mathematical precision, even if this is "incorrect". At the same time, yaidom tries to be useful in practice. For example, yaidom compromises "correctness" in the following ways:

Yaidom, and in particular the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi, eu.cdevreeze.yaidom.resolved and eu.cdevreeze.yaidom.simple sub-packages, contains the following layers:

It makes sense to read this documentation, because it helps in getting up-to-speed with yaidom.

Basic concepts

In real world XML, elements (and sometimes attributes) tend to have names within a certain namespace. There are 2 kinds of names at play here:

They are represented by immutable classes eu.cdevreeze.yaidom.core.QName and eu.cdevreeze.yaidom.core.EName, respectively.

Qualified names occur in XML, whereas expanded names do not. Yet qualified names have no meaning on their own. They need to be resolved to expanded names, via the in-scope namespaces. Note that the term "qualified name" is often used for what yaidom (and the Namespaces specification) calls "expanded name", and that most XML APIs do not distinguish between the 2 kinds of names. Yaidom has to clearly make this distinction, in order to model namespaces correctly.

To resolve qualified names to expanded names, yaidom distinguishes between:

They are represented by immutable classes eu.cdevreeze.yaidom.core.Declarations and eu.cdevreeze.yaidom.core.Scope, respectively.

Namespace declarations occur in XML, whereas in-scope namespaces do not. The latter are the accumulated effect of the namespace declarations of the element itself, if any, and those in ancestor elements.

Note: in the code examples below, we assume the following import:

import eu.cdevreeze.yaidom.core._

To see the resolution of qualified names in action, consider the following sample XML:

<book:Bookstore xmlns:book="http://bookstore/book" xmlns:auth="http://bookstore/author">
  <book:Book ISBN="978-0321356680" Price="35" Edition="2">
    <book:Title>Effective Java (2nd Edition)</book:Title>
    <book:Authors>
      <auth:Author>
        <auth:First_Name>Joshua</auth:First_Name>
        <auth:Last_Name>Bloch</auth:Last_Name>
      </auth:Author>
    </book:Authors>
  </book:Book>
  <book:Book ISBN="978-0981531649" Price="35" Edition="2">
    <book:Title>Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition</book:Title>
    <book:Authors>
      <auth:Author>
        <auth:First_Name>Martin</auth:First_Name>
        <auth:Last_Name>Odersky</auth:Last_Name>
      </auth:Author>
      <auth:Author>
        <auth:First_Name>Lex</auth:First_Name>
        <auth:Last_Name>Spoon</auth:Last_Name>
      </auth:Author>
      <auth:Author>
        <auth:First_Name>Bill</auth:First_Name>
        <auth:Last_Name>Venners</auth:Last_Name>
      </auth:Author>
    </book:Authors>
  </book:Book>
</book:Bookstore>

Consider the last element with qualified name QName("book:Book"). To resolve this qualified name as expanded name, we need to know the namespaces in scope at that element. To compute the in-scope namespaces, we need to accumulate the namespace declarations of the last book:Book element and of its ancestor element(s), starting with the root element.

The start Scope is "parent scope" Scope.Empty. Then, in the root element we find namespace declarations:

Declarations.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author")

This leads to the following namespaces in scope at the root element:

Scope.Empty.resolve(Declarations.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author"))

which is equal to:

Scope.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author")

We find no other namespace declarations in the last book:Book element or its ancestor(s), so the computed scope is also the scope of the last book:Book element.

Then QName("book:Book") is resolved as follows:

Scope.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author").resolveQNameOption(QName("book:Book"))

which is equal to:

Some(EName("{http://bookstore/book}Book"))

This namespace support in yaidom has mathematical rigor. The immutable classes QName, EName, Declarations and Scope have precise definitions, reflected in their implementations, and they obey some interesting properties. For example, if we correctly define Scope operation relativize (along with resolve), we get:

scope1.resolve(scope1.relativize(scope2)) == scope2

This may not sound like much, but by getting the basics right, yaidom succeeds in offering first-class support for XML namespaces, without the magic and namespace-related bugs often found in other XML libraries.

There are 2 other basic concepts in this package, representing paths to elements:

They are represented by immutable classes eu.cdevreeze.yaidom.core.PathBuilder and eu.cdevreeze.yaidom.core.Path, respectively.

Path builders are like canonical XPath expressions, yet they do not contain the root element itself, and indexing starts with 0 instead of 1.

For example, the last name of the first author of the last book element has path:

Path.from(
  EName("{http://bookstore/book}Book") -> 1,
  EName("{http://bookstore/book}Authors") -> 0,
  EName("{http://bookstore/author}Author") -> 0,
  EName("{http://bookstore/author}Last_Name") -> 0
)

This path could be written as path builder as follows:

PathBuilder.from(QName("book:Book") -> 1, QName("book:Authors") -> 0, QName("auth:Author") -> 0, QName("auth:Last_Name") -> 0)

Using the Scope mentioned earlier, the latter path builder resolves to the path given before that, by invoking method PathBuilder.build(scope). In order for this to work, the Scope must be invertible. That is, there must be a one-to-one correspondence between prefixes ("" for the default namespace) and namespace URIs, because otherwise the index numbers may differ. Also note that the prefixes book and auth in the path builder are arbitrary, and need not match with the prefixes used in the XML tree itself.

Uniform query API traits

Yaidom provides a relatively small query API, to query an individual element for collections of child elements, descendant elements or descendant-or-self elements. The resulting collections are immutable Scala collections, that can further be manipulated using the Scala Collections API.

This query API is uniform, in that different element implementations share (most of) the same query API. It is also element-centric (unlike standard Scala XML and Anti-XML).

For example, consider the XML example given earlier, as a Scala XML literal named bookstore. We can wrap this Scala XML Elem into a yaidom wrapper of type ScalaXmlElem, named bookstoreElem. Then we can query for all books, that is, all descendant-or-self elements with resolved (or expanded) name EName("{http://bookstore/book}Book"), as follows:

bookstoreElem filterElemsOrSelf (elem => elem.resolvedName == EName("{http://bookstore/book}Book"))

The result would be an immutable IndexedSeq of ScalaXmlElem instances, holding 2 book elements.

We could instead have written:

bookstoreElem.filterElemsOrSelf(EName("{http://bookstore/book}Book"))

with the same result, due to an implicit conversion from expanded names to element predicates.

Instead of searching for appropriate descendant-or-self elements, we could have searched for descendant elements only, without altering the result in this case:

bookstoreElem filterElems (elem => elem.resolvedName == EName("{http://bookstore/book}Book"))

or:

bookstoreElem.filterElems(EName("{http://bookstore/book}Book"))

We could even have searched for appropriate child elements only, without altering the result in this case:

bookstoreElem filterChildElems (elem => elem.resolvedName == EName("{http://bookstore/book}Book"))

or:

bookstoreElem.filterChildElems(EName("{http://bookstore/book}Book"))

or, knowing that all child elements are books:

bookstoreElem.findAllChildElems

We could find all authors of the Scala book as follows:

for {
  bookElem <- bookstoreElem filterChildElems (elem => elem.resolvedName == EName("{http://bookstore/book}Book"))
  if bookElem.attributeOption(EName("ISBN")).contains("978-0981531649")
  authorElem <- bookElem filterElems (elem => elem.resolvedName == EName("{http://bookstore/author}Author"))
} yield authorElem

or:

for {
  bookElem <- bookstoreElem.filterChildElems(EName("{http://bookstore/book}Book"))
  if bookElem.attributeOption(EName("ISBN")).contains("978-0981531649")
  authorElem <- bookElem.filterElems(EName("{http://bookstore/author}Author"))
} yield authorElem

We could even use operator notation, as follows:

for {
  bookElem <- bookstoreElem \ (elem => elem.resolvedName == EName("{http://bookstore/book}Book"))
  if (bookElem \@ EName("ISBN")).contains("978-0981531649")
  authorElem <- bookElem \\ (elem => elem.resolvedName == EName("{http://bookstore/author}Author"))
} yield authorElem

or:

for {
  bookElem <- bookstoreElem \ EName("{http://bookstore/book}Book")
  if (bookElem \@ EName("ISBN")).contains("978-0981531649")
  authorElem <- bookElem \\ EName("{http://bookstore/author}Author")
} yield authorElem

where \\ stands for filterElemsOrSelf.

There is no explicit support for filtering on the "self" element itself. In the example above, we might want to check if the root element has the expected EName, for instance. That is easy to express using a simple idiom, however. The last example then becomes:

for {
  bookstoreElem <- Vector(bookstoreElem)
  if bookstoreElem.resolvedName == EName("{http://bookstore/book}Bookstore")
  bookElem <- bookstoreElem \ EName("{http://bookstore/book}Book")
  if (bookElem \@ EName("ISBN")).contains("978-0981531649")
  authorElem <- bookElem \\ EName("{http://bookstore/author}Author")
} yield authorElem

Now suppose the same XML is stored in a (org.w3c.dom) DOM tree, wrapped in a DomElem bookstoreElem. Then the same queries would use exactly the same code as above! The result would be a collection of DomElem instances instead of ScalaXmlElem instances, however. There are many more element implementations in yaidom, and they share (most of) the same query API. Therefore this query API is called a uniform query API.

The last example, using operator notation, looks a bit more "XPath-like". It is more verbose than queries in Scala XML, however, partly because in yaidom these operators cannot be chained. Yet this is with good reason. Yaidom does not blur the distinction between elements and element collections, and therefore does not offer any XPath experience. The small price paid in verbosity is made up for by precision. The yaidom query API traits have very precise definitions of their operations, as can be seen in the corresponding documentation.

The uniform query API traits turn minimal APIs into richer APIs, where each richer API is defined very precisely in terms of the minimal API. The most important query API trait is eu.cdevreeze.yaidom.queryapi.ElemLike. It needs to be given a method implementation to query for child elements (not child nodes in general, but just child elements!), and it offers methods to query for some or all child elements, descendant elements, and descendant-or-self elements. That is, the minimal API consists of abstract method findAllChildElems, and it offers methods such as filterChildElems, filterElems and filterElemsOrSelf. This trait has no knowledge about elements at all, other than the fact that elements can have child elements.

Trait eu.cdevreeze.yaidom.queryapi.HasEName needs minimal knowledge about elements themselves, viz. that elements have a "resolved" (or expanded) name, and "resolved" attributes (mapping attribute expanded names to attribute values). That is, it needs to be given implementations of abstract methods resolvedName and resolvedAttributes, and then offers methods to query for individual attributes or the local name of the element.

It is important to note that yaidom does not consider namespace declarations to be attributes themselves. Otherwise, there would have been circular dependencies between both concepts, because attributes with namespaces require in-scope namespaces and therefore namespace declarations for resolving the names of these attributes.

Many traits, such as eu.cdevreeze.yaidom.queryapi.HasEName, are just "capabilities", and need to be combined with trait eu.cdevreeze.yaidom.queryapi.ElemLike in order to offer a useful element querying API.

Note that trait eu.cdevreeze.yaidom.queryapi.ElemLike only knows about elements, not about other kinds of nodes. Of course the actual element implementations mixing in this query API know about other node types, but that knowledge is outside the uniform query API. Note that the example queries above only use the minimal element knowledge that traits ElemLike and HasEName together have about elements. Therefore the query code can be used unchanged for different element implementations.

Trait eu.cdevreeze.yaidom.queryapi.IsNavigable is used to navigate to an element given a Path.

Trait eu.cdevreeze.yaidom.queryapi.UpdatableElemLike (which extends trait IsNavigable) offers functional updates at given paths. Whereas the traits mentioned above know only about elements, this trait knows that elements have some node super-type.

Instead of functional updates at given paths, elements can also be "transformed" functionally without specifying any paths. This is offered by trait eu.cdevreeze.yaidom.queryapi.TransformableElemLike. The Scala XML and DOM wrappers above do not mix in this trait.

Some element implementations

The uniform query API traits, especially ElemLike and its sub-trait ElemLike are mixed in by many element implementations. In package simple there are 2 immutable element implementations, eu.cdevreeze.yaidom.simple.ElemBuilder and eu.cdevreeze.yaidom.simple.Elem.

Class eu.cdevreeze.yaidom.simple.Elem is the default element implementation of yaidom. It extends class eu.cdevreeze.yaidom.simple.Node. The latter also has sub-classes for text nodes, comments, entity references and processing instructions. Class eu.cdevreeze.yaidom.simple.Document contains a document Elem, but is not a Node sub-class itself.

The eu.cdevreeze.yaidom.simple.Elem class has the following characteristics:

Creating such Elem trees by hand is a bit cumbersome, partly because scopes have to be passed to each Elem in the tree. The latter is not needed if we use class eu.cdevreeze.yaidom.simple.ElemBuilder to create element trees by hand. When the tree has been fully created as ElemBuilder, invoke method ElemBuilder.build(parentScope) to turn it into an Elem.

Like their super-classes Node and NodeBuilder, classes Elem and ElemBuilder have very much in common. Both are immutable, easy to compose (ElemBuilder instances even more so), equality is reference equality, etc. The most important differences are as follows:

The Effective Java book element in the XML example above could have been written as ElemBuilder (without the inter-element whitespace) as follows:

import NodeBuilder._

elem(
  qname = QName("book:Book"),
  attributes = Vector(QName("ISBN") -> "978-0321356680", QName("Price") -> "35", QName("Edition") -> "2"),
  children = Vector(
    elem(
      qname = QName("book:Title"),
      children = Vector(
        text("Effective Java (2nd Edition)")
      )
    ),
    elem(
      qname = QName("book:Authors"),
      children = Vector(
        elem(
          qname = QName("auth:Author"),
          children = Vector(
            elem(
              qname = QName("auth:First_Name"),
              children = Vector(
                text("Joshua")
              )
            ),
            elem(
              qname = QName("auth:Last_Name"),
              children = Vector(
                text("Bloch")
              )
            )
          )
        )
      )
    )
  )
)

This ElemBuilder (say, eb) lacks namespace declarations for prefixes book and auth. So, the following returns false:

eb.canBuild(Scope.Empty)

while the following returns true:

eb.canBuild(Scope.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author"))

Indeed,

eb.build(Scope.from("book" -> "http://bookstore/book", "auth" -> "http://bookstore/author"))

returns the element tree as Elem.

Note that the distinction between ElemBuilder and Elem "solves" the mismatch that immutable ("functional") element trees are constructed in a bottom-up manner, while namespace scoping works in a top-down manner. (See also Anti-XML issue 78, in https://github.com/djspiewak/anti-xml/issues/78).

There are many more element implementations in yaidom, most of them in sub-packages of this package. Yaidom is extensible in that new element implementations can be invented, for example elements that are better "roundtrippable" (at the expense of "composability"), or yaidom wrappers around other DOM-like APIs (such as XOM or JDOM2). The current element implementations in yaidom are for example:

This illustrates that especially trait ElemLike is a uniform query API in yaidom.

One yaidom wrapper that is very useful is a Saxon tiny tree yaidom wrapper. It has not been provided out of the box, in order to prevent any yaidom dependency on Saxon (which could have been an optional dependency), and because of changes among Saxon versions. Nevertheless, a Saxon wrapper has been tried out in test code (just like wrappers around XOM and JDOM), thus showing how easy it is to create such a wrapper ourselves. In the case of Saxon, when using schema-awareness (Saxon-EE), the wrapped tiny tree would contain interesting type information.

Packages and dependencies

Yaidom has the following packages, and layering between packages (mentioning the lowest layers first):

Indeed, all yaidom package dependencies are uni-directional.

Notes on performance

Yaidom can be quite memory-hungry. One particular cause of that is the possible creation of very many duplicate EName and QName instances. This can be the case while parsing XML into yaidom documents, or while querying yaidom element trees.

The user of the library can reduce memory consumption to a large extent, and yaidom facilitates that.

As for querying, prefer:

import HasENameApi._

bookstoreElem filterElemsOrSelf withEName("http://bookstore/book", "Book")

to:

bookstoreElem.filterElemsOrSelf(EName("http://bookstore/book", "Book"))

to avoid unnecessary (large scale) EName object creation.

To reduce the memory footprint of parsed XML trees, see eu.cdevreeze.yaidom.core.ENameProvider and eu.cdevreeze.yaidom.core.QNameProvider.

For example, during the startup phase of an application, we could set the global ENameProvider as follows:

ENameProvider.globalENameProvider.become(new ENameProvider.ENameProviderUsingImmutableCache(knownENames))

Note that the global ENameProvider or QNameProvider can typically be configured rather late during development, but the memory cost savings can be substantial. Also note that the global ENameProvider or QNameProvider can be used implicitly in application code, by writing:

bookstoreElem filterElemsOrSelf getEName("http://bookstore/book", "Book")

using an implicit ENameProvider, whose members are in scope. Still, for querying the first alternative using withEName is better, but there are likely many scenarios in yaidom client code where an implicit ENameProvider or QNameProvider makes sense.

The bottom line is that yaidom can be configured to be far less memory-hungry, and that yaidom client code can also take some responsibility in reducing memory usage.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. yaidom
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. package convert

    Permalink

    Support for conversions from/to yaidom.

    Support for conversions from/to yaidom. This package mostly contains conversions between yaidom objects and JAXP DOM or StAX objects, in both directions.

    This conversion support is used by the Document parsers and printers in the parse and print packages, respectively. This package can also be used directly by consumers of the yaidom API.

    These JAXP-object conversions suggest that yaidom is optimistic about the available (heap) memory.

    This package depends on the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi and eu.cdevreeze.yaidom.simple packages, and not the other way around. The eu.cdevreeze.yaidom.parse and eu.cdevreeze.yaidom.print packages depend on this package.

  2. package core

    Permalink

    This package contains the core concepts, such as qualified names, expanded names, namespace declarations, in-scope namespaces, paths and path builders.

    This package contains the core concepts, such as qualified names, expanded names, namespace declarations, in-scope namespaces, paths and path builders.

    This package depends on no other packages in yaidom, but almost all other packages do depend on this one.

  3. package indexed

    Permalink

    This package contains element representations that contain the "context" of the element.

    This package contains element representations that contain the "context" of the element. That is, the elements in this package are pairs of a root element and a path (to the actual element itself). The "context" of an element also contains an optional document URI.

    An example of where such a representation can be useful is XML Schema. After all, to interpret an element definition in an XML schema, we need context of the element definition to determine the target namespace, or to determine whether the element definition is top level, etc.

    Below follows a simple example query, using the uniform query API:

    // Note the import of package indexed, and not of its members. That is indeed a best practice!
import eu.cdevreeze.yaidom.indexed

val indexedBookstoreElem = indexed.Elem(bookstoreElem)

val scalaBookAuthors =
  for {
    bookElem <- indexedBookstoreElem \ EName("{http://bookstore/book}Book")
    if (bookElem \@ EName("ISBN")).contains("978-0981531649")
    authorElem <- bookElem \\ EName("{http://bookstore/author}Author")
  } yield authorElem

    The query for Scala book authors would have been exactly the same if normal Elems had been used instead of indexed.Elems (replacing indexedBookstoreElem by bookstoreElem)!

    There is no explicit functional update support for the indexed elements in this package. Of course the underlying elements can be functionally updated (for element implementations that offer such update support), and indexed elements can be created from the update results, but this is hardly efficient functional update support.

    One problem with efficient functional updates for indexed elements is that updating just one child element means that all subsequent child elements may have to be updated as well, adapting the stored paths. In comparison, simple elements do not have this restriction, and can be updated in isolation. Hence the functional update support for simple elements but not for the different indexed element implementations.

  4. package jsdom

    Permalink

    Wrapper around class org.scalajs.dom.raw.Element, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.

    Wrapper around class org.scalajs.dom.raw.Element, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.

    It is comparable to the yaidom DOM wrapper on the JVM, except that this (browser) DOM runs in the browser.

  5. package queryapi

    Permalink

    This package contains the (renewed) query API traits.

    This package contains the (renewed) query API traits. It contains both the purely abstract API traits as well as the partial implementation traits.

    Generic code abstracting over yaidom element implementations should either use trait ClarkElemApi or sub-trait ScopedElemApi, or even BackingElemApi, depending on the abstraction level.

    Most API traits are orthogonal, but some API traits are useful combinations of other ones. Examples include the above-mentioned ClarkElemApi and ScopedElemApi traits.

    Simplicity and consistency of the entire query API are 2 important design considerations. For example, the query API methods themselves use no parameterized types. Note how the resulting API with type members is essentially the same as the "old" yaidom query API using type parameters, except that the purely abstract traits are less constrained in the type members.

    This package depends only on the core package in yaidom, but many other packages do depend on this one.

    Note: whereas the old query API used F-bounded polymorphism with type parameters extensively, this new query API essentially just uses type member ThisElem, defined in a common super-trait. The old query API may be somewhat easier to develop (that is, convincing the compiler), but the new query API is easier to use as generic "backend" element query API. As an example, common "bridge" element query APIs come to mind, used within type-safe XML dialect DOM tree implementations. The reason this is easier with the new API is intuitively that fewer type constraints leak to the query API client code.

  6. package resolved

    Permalink

    This package contains element representations that can be compared for (some notion of "value") equality, unlike normal yaidom nodes.

    This package contains element representations that can be compared for (some notion of "value") equality, unlike normal yaidom nodes. That notion of equality is simple to understand, but "naive". The user is of the API must take control over what is compared for equality.

    See eu.cdevreeze.yaidom.resolved.Node for why this package is named resolved.

    The most important difference with normal Elems is that qualified names do not occur, but only expanded (element and attribute) names. This reminds of James Clark notation for XML trees and expanded names, where qualified names are absent.

    Moreover, the only nodes in this package are element and text nodes.

    Below follows a simple example query, using the uniform query API:

    // Note the import of package resolved, and not of its members. That is indeed a best practice!
import eu.cdevreeze.yaidom.resolved

val resolvedBookstoreElem = resolved.Elem(bookstoreElem)

val scalaBookAuthors =
  for {
    bookElem <- resolvedBookstoreElem \ EName("{http://bookstore/book}Book")
    if (bookElem \@ EName("ISBN")).contains("978-0981531649")
    authorElem <- bookElem \\ EName("{http://bookstore/author}Author")
  } yield authorElem

    The query for Scala book authors would have been exactly the same if normal Elems had been used instead of resolved.Elems (replacing resolvedBookstoreElem by bookstoreElem)!

  7. package simple

    Permalink

    This package contains the default element implementation.

    This package contains the default element implementation.

    This package depends only on the core and queryapi packages in yaidom, but many other packages do depend on this one.

  8. package utils

    Permalink

    Several utilities, such as custom ENameProviders and QNameProviders.

    Several utilities, such as custom ENameProviders and QNameProviders. They are utilities "on top of yaidom", so the rest of yaidom has no dependencies on this package, but this package does depend on the rest of yaidom.

Inherited from AnyRef

Inherited from Any

Ungrouped