Support for conversions from/to yaidom.
Support for conversions from/to yaidom. This package mostly contains conversions between yaidom objects and JAXP DOM or StAX objects, in both directions.
This conversion support is used by the Document parsers and printers in the parse
and print
packages, respectively.
This package can also be used directly by consumers of the yaidom API.
These JAXP-object conversions suggest that yaidom is optimistic about the available (heap) memory.
This package depends on the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi and eu.cdevreeze.yaidom.simple packages, and not the other way around. The eu.cdevreeze.yaidom.parse and eu.cdevreeze.yaidom.print packages depend on this package.
This package contains the core concepts, such as qualified names, expanded names, namespace declarations, in-scope namespaces, paths and path builders.
This package contains the core concepts, such as qualified names, expanded names, namespace declarations, in-scope namespaces, paths and path builders.
This package depends on no other packages in yaidom, but almost all other packages do depend on this one.
Wrapper around class org.w3c.dom.Element
, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.
Wrapper around class org.w3c.dom.Element
, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.
This wrapper is not thread-safe, and should only be used if the immutable element classes such as eu.cdevreeze.yaidom.simple.Elem are not the best fit.
Such scenarios could be as follows:
Yet be aware that the advantages of immutability and thread-safety (offered by immutable Elem
classes) are lost when using
this wrapper API. Mutable DOM trees are also very easy to break, even via the ElemLike
API, if element predicates with
side-effects are used.
To explain the "round-tripping" item above, note that class eu.cdevreeze.yaidom.simple.Elem considers attributes in an element unordered,
let alone namespace declarations. That is consistent with the XML Infoset specification, but can sometimes be impractical.
Using org.w3c.dom.Element
instances, parsed from XML input sources, chances are that this order is retained.
There are of course limitations to what formatting data is retained in a DOM tree. A good example is the short versus long form of an empty element. Typically parsers do not pass any information about this distinction, so it is unknown whether the XML input source used the long or short form for an empty element.
It should also be noted that the configuration of XML parsers and serializers can be of substantial influence on the extent that "round-tripping" keeps the XML string the same. Whitespace handling is one such area in which different configurations can lead to quite different "round-tripping" results.
Note that in one way these wrappers are somewhat unnatural: the ElemLike
API uses immutable Scala collections everywhere,
whereas the elements of those collections are mutable (!) DOM node wrappers. The wrappers are idiomatic Scala in their use of
the Scala Collections API, whereas the wrapped DOM nodes come from a distant past, when imperative programming and "mutability
everywhere" ruled.
In comparison to XPath against DOM trees, the ElemLike
API may be more verbose, but it requires no setup and
"result set handling" boilerplate.
This package contains element representations that contain the "context" of the element.
This package contains element representations that contain the "context" of the element. That is, the elements in this package are pairs of a root element and a path (to the actual element itself). The "context" of an element also contains an optional document URI.
An example of where such a representation can be useful is XML Schema. After all, to interpret an element definition in an XML schema, we need context of the element definition to determine the target namespace, or to determine whether the element definition is top level, etc.
Below follows a simple example query, using the uniform query API:
// Note the import of package indexed, and not of its members. That is indeed a best practice! import eu.cdevreeze.yaidom.indexed val indexedBookstoreElem = indexed.Elem(bookstoreElem) val scalaBookAuthors = for { bookElem <- indexedBookstoreElem \ EName("{http://bookstore/book}Book") if (bookElem \@ EName("ISBN")) == Some("978-0981531649") authorElem <- bookElem \\ EName("{http://bookstore/author}Author") } yield authorElem
The query for Scala book authors would have been exactly the same if normal Elem
s had been used instead of indexed.Elem
s
(replacing indexedBookstoreElem
by bookstoreElem
)!
Support for parsing XML into yaidom Document
s and Elem
s.
Support for parsing XML into yaidom Document
s and Elem
s. This package offers the eu.cdevreeze.yaidom.parse.DocumentParser
trait, as well as several implementations. Those implementations use JAXP (SAX, DOM or StAX), and most of them use the convert
package
to convert JAXP artifacts to yaidom Document
s.
For example:
val docParser = DocumentParserUsingSax.newInstance() val doc: Document = docParser.parse(docUri)
This example chose a SAX-based implementation, and used the default configuration of that document parser.
Having several different fully configurable JAXP-based implementations shows that yaidom is pessimistic about the transparency of parsing and
printing XML. It also shows that yaidom is optimistic about the available (heap) memory and processing power, because of the 2 separated
steps of JAXP parsing/printing and (in-memory) convert
conversions. Using JAXP means that escaping of characters is something
that JAXP deals with, and that's definitely better than trying to do it yourself.
One DocumentParser
implementation does not use any convert
conversion. That is DocumentParserUsingSax
. It is likely the
fastest of the DocumentParser
implementations.
The preferred DocumentParser
for XML (not HTML) parsing is DocumentParserUsingDomLS
, if memory usage is not an
issue. This DocumentParser
implementation is best integrated with DOM, and is highly configurable, although
DOM LS configuration is somewhat involved.
This package depends on the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi, eu.cdevreeze.yaidom.simple and eu.cdevreeze.yaidom.convert packages, and not the other way around.
Support for "printing" yaidom Document
s and Elem
s.
Support for "printing" yaidom Document
s and Elem
s. This package offers the eu.cdevreeze.yaidom.print.DocumentPrinter
trait, as well as several implementations. Most of those implementations use the convert
package to convert yaidom Document
s to
JAXP artifacts, and all use JAXP (DOM, SAX or StAX).
For example:
val docPrinter = DocumentPrinterUsingSax.newInstance() docPrinter.print(doc, "UTF-8", System.out)
This example chose a SAX-based implementation, and used the default configuration of that document printer.
Having several different fully configurable JAXP-based implementations shows that yaidom is pessimistic about the transparency of parsing and
printing XML. It also shows that yaidom is optimistic about the available (heap) memory and processing power, because of the 2 separated
steps of JAXP parsing/printing and (in-memory) convert
conversions. Using JAXP means that escaping of characters is something
that JAXP deals with, and that's definitely better than trying to do it yourself.
One DocumentPrinter
implementation does not use any convert
conversion. That is DocumentPrinterUsingSax
. It is likely the
fastest of the DocumentPrinter
implementations, as well as the one using the least memory.
The preferred DocumentPrinter
for XML (not HTML) printing is DocumentPrinterUsingDomLS
, if memory usage is not an
issue. This DocumentPrinter
implementation is best integrated with DOM, and is highly configurable, although
DOM LS configuration is somewhat involved.
This package depends on the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi, eu.cdevreeze.yaidom.simple and eu.cdevreeze.yaidom.convert packages, and not the other way around.
This package contains the query API traits.
This package contains the query API traits. It contains both the purely abstract API traits as well as the partial implementation traits.
Generic code abstracting over yaidom element implementations should either use
trait ClarkElemApi
or sub-trait ScopedElemApi
, depending on the abstraction level.
Most API traits are orthogonal, but some API traits are useful combinations of other ones. Examples include
the above-mentioned ClarkElemApi
and ScopedElemApi
traits.
This package depends only on the core package in yaidom, but many other packages do depend on this one.
This package contains element representations that can be compared for (some notion of "value") equality, unlike normal yaidom nodes.
This package contains element representations that can be compared for (some notion of "value") equality, unlike normal yaidom nodes. That notion of equality is simple to understand, but "naive". The user is of the API must take control over what is compared for equality.
See eu.cdevreeze.yaidom.resolved.Node for why this package is named resolved
.
The most important difference with normal Elem
s is that qualified names do not occur,
but only expanded (element and attribute) names. This reminds of James Clark notation for XML trees and
expanded names, where qualified names are absent.
Moreover, the only nodes in this package are element and text nodes.
Below follows a simple example query, using the uniform query API:
// Note the import of package resolved, and not of its members. That is indeed a best practice! import eu.cdevreeze.yaidom.resolved val resolvedBookstoreElem = resolved.Elem(bookstoreElem) val scalaBookAuthors = for { bookElem <- resolvedBookstoreElem \ EName("{http://bookstore/book}Book") if (bookElem \@ EName("ISBN")) == Some("978-0981531649") authorElem <- bookElem \\ EName("{http://bookstore/author}Author") } yield authorElem
The query for Scala book authors would have been exactly the same if normal Elem
s had been used instead of resolved.Elem
s
(replacing resolvedBookstoreElem
by bookstoreElem
)!
Wrapper around class scala.xml.Elem
, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.
Wrapper around class scala.xml.Elem
, adapting it to the eu.cdevreeze.yaidom.queryapi.ElemLike API.
This wrapper brings the uniform yaidom query API to Scala XML literals (and Scala XML Elems in general).
For some namespace-related pitfalls and peculiarities, see eu.cdevreeze.yaidom.scalaxml.ScalaXmlElem.
This package contains the default element implementation.
This package contains the default element implementation.
This package depends only on the core and queryapi packages in yaidom, but many other packages do depend on this one.
Several utilities, such as custom ENameProviders and QNameProviders.
Several utilities, such as custom ENameProviders and QNameProviders. They are utilities "on top of yaidom", so the rest of yaidom has no dependencies on this package, but this package does depend on the rest of yaidom.
Yaidom is yet another Scala immutable DOM-like XML API. Some other well-known Scala immutable DOM-like APIs are the standard scala.xml API and Anti-XML. The latter API is considered by many to be an improvement over the former, but both APIs:
Yaidom takes a different approach, avoiding XPath-like query support, and offering good namespace and decent (functional) update support. Yaidom is also characterized by almost mathematical precision and clarity. Still, the API remains practical and pragmatic. In particular, the API user has much configuration control over parsing and serialization, because yaidom exposes the underlying JAXP parsers and serializers, which can be configured by the library user.
Yaidom chooses its battles. For example, given that DTDs do not know about namespaces, yaidom offers good namespace support, but ignores DTDs entirely. Of course the underlying XML parser may still validate XML against a DTD, if so desired. As another example, yaidom tries to leave the handling of the gory details of XML processing (such as whitespace handling) as much as possible to JAXP (and JAXP parser/serializer configuration). As yet another example, yaidom knows nothing about (XML Schema) types of elements and attributes.
As mentioned above, yaidom tries to treat basic XML processing with almost mathematical precision, even if this is "incorrect". At the same time, yaidom tries to be useful in practice. For example, yaidom compromises "correctness" in the following ways:
Yaidom, and in particular the eu.cdevreeze.yaidom.core, eu.cdevreeze.yaidom.queryapi, eu.cdevreeze.yaidom.resolved and eu.cdevreeze.yaidom.simple sub-packages, contains the following layers:
core
package)queryapi
package)resolved
andsimple
packages)It makes sense to read this documentation, because it helps in getting up-to-speed with yaidom.
Basic concepts
In real world XML, elements (and sometimes attributes) tend to have names within a certain namespace. There are 2 kinds of names at play here:
book:Title
, and unprefixed names, such asEdition
{http://bookstore/book}Title
(in James Clark notation), and not having a namespace, such asEdition
They are represented by immutable classes eu.cdevreeze.yaidom.core.QName and eu.cdevreeze.yaidom.core.EName, respectively.
Qualified names occur in XML, whereas expanded names do not. Yet qualified names have no meaning on their own. They need to be resolved to expanded names, via the in-scope namespaces. Note that the term "qualified name" is often used for what yaidom (and the Namespaces specification) calls "expanded name", and that most XML APIs do not distinguish between the 2 kinds of names. Yaidom has to clearly make this distinction, in order to model namespaces correctly.
To resolve qualified names to expanded names, yaidom distinguishes between:
They are represented by immutable classes eu.cdevreeze.yaidom.core.Declarations and eu.cdevreeze.yaidom.core.Scope, respectively.
Namespace declarations occur in XML, whereas in-scope namespaces do not. The latter are the accumulated effect of the namespace declarations of the element itself, if any, and those in ancestor elements.
Note: in the code examples below, we assume the following import:
import eu.cdevreeze.yaidom.core._
To see the resolution of qualified names in action, consider the following sample XML:
Consider the last element with qualified name
QName("book:Book")
. To resolve this qualified name as expanded name, we need to know the namespaces in scope at that element. To compute the in-scope namespaces, we need to accumulate the namespace declarations of the lastbook:Book
element and of its ancestor element(s), starting with the root element.The start Scope is "parent scope"
Scope.Empty
. Then, in the root element we find namespace declarations:This leads to the following namespaces in scope at the root element:
which is equal to:
We find no other namespace declarations in the last
book:Book
element or its ancestor(s), so the computed scope is also the scope of the lastbook:Book
element.Then
QName("book:Book")
is resolved as follows:which is equal to:
This namespace support in yaidom has mathematical rigor. The immutable classes
QName
,EName
,Declarations
andScope
have precise definitions, reflected in their implementations, and they obey some interesting properties. For example, if we correctly define Scope operationrelativize
(along withresolve
), we get:This may not sound like much, but by getting the basics right, yaidom succeeds in offering first-class support for XML namespaces, without the magic and namespace-related bugs often found in other XML libraries.
There are 2 other basic concepts in this package, representing paths to elements:
They are represented by immutable classes eu.cdevreeze.yaidom.core.PathBuilder and eu.cdevreeze.yaidom.core.Path, respectively.
Path builders are like canonical XPath expressions, yet they do not contain the root element itself, and indexing starts with 0 instead of 1.
For example, the last name of the first author of the last book element has path:
This path could be written as path builder as follows:
Using the Scope mentioned earlier, the latter path builder resolves to the path given before that, by invoking method
PathBuilder.build(scope)
. In order for this to work, the Scope must be invertible. That is, there must be a one-to-one correspondence between prefixes ("" for the default namespace) and namespace URIs, because otherwise the index numbers may differ. Also note that the prefixesbook
andauth
in the path builder are arbitrary, and need not match with the prefixes used in the XML tree itself.Uniform query API traits
Yaidom provides a relatively small query API, to query an individual element for collections of child elements, descendant elements or descendant-or-self elements. The resulting collections are immutable Scala collections, that can further be manipulated using the Scala Collections API.
This query API is uniform, in that different element implementations share (most of) the same query API. It is also element-centric (unlike standard Scala XML and Anti-XML).
For example, consider the XML example given earlier, as a Scala XML literal named
bookstore
. We can wrap this Scala XML Elem into a yaidom wrapper of type eu.cdevreeze.yaidom.scalaxml.ScalaXmlElem, namedbookstoreElem
. Then we can query for all books, that is, all descendant-or-self elements with resolved (or expanded) nameEName("{http://bookstore/book}Book")
, as follows:The result would be an immutable IndexedSeq of
ScalaXmlElem
instances, holding 2 book elements.We could instead have written:
bookstoreElem.filterElemsOrSelf(EName("{http://bookstore/book}Book"))
with the same result, due to an implicit conversion from expanded names to element predicates.
Instead of searching for appropriate descendant-or-self elements, we could have searched for descendant elements only, without altering the result in this case:
or:
bookstoreElem.filterElems(EName("{http://bookstore/book}Book"))
We could even have searched for appropriate child elements only, without altering the result in this case:
or:
bookstoreElem.filterChildElems(EName("{http://bookstore/book}Book"))
or, knowing that all child elements are books:
We could find all authors of the Scala book as follows:
or:
We could even use operator notation, as follows:
or:
where
\\
stands forfilterElemsOrSelf
.There is no explicit support for filtering on the "self" element itself. In the example above, we might want to check if the root element has the expected EName, for instance. That is easy to express using a simple idiom, however. The last example then becomes:
Now suppose the same XML is stored in a (org.w3c.dom) DOM tree, wrapped in a eu.cdevreeze.yaidom.dom.DomElem
bookstoreElem
. Then the same queries would use exactly the same code as above! The result would be a collection ofDomElem
instances instead ofScalaXmlElem
instances, however. There are many more element implementations in yaidom, and they share (most of) the same query API. Therefore this query API is called a uniform query API.The last example, using operator notation, looks a bit more "XPath-like". It is more verbose than queries in Scala XML, however, partly because in yaidom these operators cannot be chained. Yet this is with good reason. Yaidom does not blur the distinction between elements and element collections, and therefore does not offer any XPath experience. The small price paid in verbosity is made up for by precision. The yaidom query API traits have very precise definitions of their operations, as can be seen in the corresponding documentation.
The uniform query API traits turn minimal APIs into richer APIs, where each richer API is defined very precisely in terms of the minimal API. The most important query API trait is eu.cdevreeze.yaidom.queryapi.ElemLike. It needs to be given a method implementation to query for child elements (not child nodes in general, but just child elements!), and it offers methods to query for some or all child elements, descendant elements, and descendant-or-self elements. That is, the minimal API consists of abstract method
findAllChildElems
, and it offers methods such asfilterChildElems
,filterElems
andfilterElemsOrSelf
. This trait has no knowledge about elements at all, other than the fact that elements can have child elements.Trait eu.cdevreeze.yaidom.queryapi.HasEName needs minimal knowledge about elements themselves, viz. that elements have a "resolved" (or expanded) name, and "resolved" attributes (mapping attribute expanded names to attribute values). That is, it needs to be given implementations of abstract methods
resolvedName
andresolvedAttributes
, and then offers methods to query for individual attributes or the local name of the element.It is important to note that yaidom does not consider namespace declarations to be attributes themselves. Otherwise, there would have been circular dependencies between both concepts, because attributes with namespaces require in-scope namespaces and therefore namespace declarations for resolving the names of these attributes.
Many traits, such as eu.cdevreeze.yaidom.queryapi.HasEName, are just "capabilities", and need to be combined with trait eu.cdevreeze.yaidom.queryapi.ElemLike in order to offer a useful element querying API.
Note that trait eu.cdevreeze.yaidom.queryapi.ElemLike only knows about elements, not about other kinds of nodes. Of course the actual element implementations mixing in this query API know about other node types, but that knowledge is outside the uniform query API. Note that the example queries above only use the minimal element knowledge that traits
ElemLike
andHasEName
together have about elements. Therefore the query code can be used unchanged for different element implementations.Trait eu.cdevreeze.yaidom.queryapi.IsNavigable is used to navigate to an element given a Path.
Trait eu.cdevreeze.yaidom.queryapi.UpdatableElemLike (which extends trait
IsNavigable
) offers functional updates at given paths. Whereas the traits mentioned above know only about elements, this trait knows that elements have some node super-type.Instead of functional updates at given paths, elements can also be "transformed" functionally without specifying any paths. This is offered by trait eu.cdevreeze.yaidom.queryapi.TransformableElemLike. The Scala XML and DOM wrappers above do not mix in this trait.
Some element implementations
The uniform query API traits, especially
ElemLike
and its sub-traitElemLike
are mixed in by many element implementations. In this package there are 2 immutable element implementations, eu.cdevreeze.yaidom.simple.ElemBuilder and eu.cdevreeze.yaidom.simple.Elem.Class eu.cdevreeze.yaidom.simple.Elem is the default element implementation of yaidom. It extends class eu.cdevreeze.yaidom.simple.Node. The latter also has sub-classes for text nodes, comments, entity references and processing instructions. Class eu.cdevreeze.yaidom.simple.Document contains a document
Elem
, but is not aNode
sub-class itself.The eu.cdevreeze.yaidom.simple.Elem class has the following characteristics:
HasParent
Scope
, but noDeclarations
parse
andprint
offerDocumentParser
andDocumentPrinter
classes for parsing/serializing these defaultElem
(andDocument
) instancesCreating such
Elem
trees by hand is a bit cumbersome, partly because scopes have to be passed to eachElem
in the tree. The latter is not needed if we use class eu.cdevreeze.yaidom.simple.ElemBuilder to create element trees by hand. When the tree has been fully created asElemBuilder
, invoke methodElemBuilder.build(parentScope)
to turn it into anElem
.Like their super-classes
Node
andNodeBuilder
, classesElem
andElemBuilder
have very much in common. Both are immutable, easy to compose (ElemBuilder
instances even more so), equality is reference equality, etc. The most important differences are as follows:Scope
, anElemBuilder
contains aDeclarations
ElemBuilder
easier to compose than anElem
, because no Scope needs to be passed around throughout the treeElemBuilder
uses a minimal query API, mixing in almost only traitsElemLike
andTransformableElemLike
ElemBuilder
neither keeps nor knows about Scopes, so does not know about resolved element/attribute namesThe Effective Java book element in the XML example above could have been written as
ElemBuilder
(without the inter-element whitespace) as follows:This
ElemBuilder
(say,eb
) lacks namespace declarations for prefixesbook
andauth
. So, the following returnsfalse
:while the following returns
true
:Indeed,
returns the element tree as
Elem
.Note that the distinction between
ElemBuilder
andElem
"solves" the mismatch that immutable ("functional") element trees are constructed in a bottom-up manner, while namespace scoping works in a top-down manner. (See also Anti-XML issue 78, in https://github.com/djspiewak/anti-xml/issues/78).There are many more element implementations in yaidom, most of them in sub-packages of this package. Yaidom is extensible in that new element implementations can be invented, for example elements that are better "roundtrippable" (at the expense of "composability"), or yaidom wrappers around other DOM-like APIs (such as XOM or JDOM2). The current element implementations in yaidom are:
Elem
by hand. See above.ElemLike
query API, but knows its ancestry, despite being immutable! This element implementation is handy for querying XML schemas, for example, because in schemas the ancestry of queried elements typically matters.ElemLike
query API.ElemLike
andHasParent
query APIs.This illustrates that especially trait
ElemLike
is a uniform query API in yaidom.One yaidom wrapper that is very useful is a Saxon tiny tree yaidom wrapper. It has not been provided out of the box, in order to prevent any yaidom dependency on Saxon (which could have been an optional dependency), and because of changes among Saxon versions. Nevertheless, a Saxon wrapper has been tried out in test code (just like wrappers around XOM and JDOM), thus showing how easy it is to create such a wrapper ourselves. In the case of Saxon, when using schema-awareness (Saxon-EE), the wrapped tiny tree would contain interesting type information.
Packages and dependencies
Yaidom has the following packages, and layering between packages (mentioning the lowest layers first):
core
package.core
andqueryapi
packages.core
,queryapi
andresolved
packages.core
,queryapi
,resolved
andsimple
packages.indexed
package, theconvert
package depends on the yaidomcore
,queryapi
,resolved
andsimple
packages.indexed
.utils
), such as eu.cdevreeze.yaidom.dom and eu.cdevreeze.yaidom.scalaxml. They depend on (some of) the packages mentioned above, but not on each other.Indeed, all yaidom package dependencies are uni-directional.
Notes on performance
Yaidom can be quite memory-hungry. One particular cause of that is the possible creation of very many duplicate EName and QName instances. This can be the case while parsing XML into yaidom documents, or while querying yaidom element trees.
The user of the library can reduce memory consumption to a large extent, and yaidom facilitates that.
As for querying, prefer:
to:
to avoid unnecessary (large scale) EName object creation.
To reduce the memory footprint of parsed XML trees, see eu.cdevreeze.yaidom.core.ENameProvider and eu.cdevreeze.yaidom.core.QNameProvider.
For example, during the startup phase of an application, we could set the global ENameProvider as follows:
ENameProvider.globalENameProvider.become(new ENameProvider.ENameProviderUsingImmutableCache(knownENames))
Note that the global ENameProvider or QNameProvider can typically be configured rather late during development, but the memory cost savings can be substantial. Also note that the global ENameProvider or QNameProvider can be used implicitly in application code, by writing:
using an implicit ENameProvider, whose members are in scope. Still, for querying the first alternative using
withEName
is better, but there are likely many scenarios in yaidom client code where an implicit ENameProvider or QNameProvider makes sense.The bottom line is that yaidom can be configured to be far less memory-hungry, and that yaidom client code can also take some responsibility in reducing memory usage.