- Reference manual
- SWI-Prolog SGML/XML parser
- Bluffer's Guide
- Predicate Reference
- Stream encoding issues
- library(xpath): Select nodes in an XML DOM
- Processing Indexed Files
- External entities
- library(pwp): Prolog Well-formed Pages
- Writing markup
- Unsupported SGML features
- SWI-Prolog SGML/XML parser
SGML or XML files are loaded through the common predicate load_structure/3. This is a predicate with many options. For simplicity a number of commonly used shorthands are provided: load_sgml_file/2, load_xml_file/2, and load_html_file/2.
- load_structure(+Source, -ListOfContent, +Options)
- Parse Source and return the resulting structure in
ListOfContent. Source is either a term of the
stream(StreamHandle)or a file-name. Options is a list of options controlling the conversion process.
A proper XML document contains only a single toplevel element whose name matches the document type. Nevertheless, a list is returned for consistency with the representation of element content. The ListOfContent consists of the following types:
- Atoms are used to represent
CDATA. Note this is possible in SWI-Prolog, as there is no length-limit on atoms and atom garbage collection is provided.
- element(Name, ListAttributes, ListOfContent)
- Name is the name of the element. Using SGML, which is
case-insensitive, all element names are returned as lowercase atoms.
ListOfAttributes is a list of Name=Value pairs for attributes. Attributes of type
CDATAare returned literal. Multi-valued attributes (
NAMES, etc.) are returned as a list of atoms. Handling attributes of the types
NUMBERSdepends on the setting of the
number(+NumberMode)attribute through set_sgml_parser/2 or load_structure/3. By default they are returned as atoms, but automatic conversion to Prolog integers is supported. ListOfContent defines the content for the element.
- If an entity with declared content-type
SDATAis encountered, this term is returned holding the data in Text.
- If an entity with declared content-type
NDATAis encountered, this term is returned holding the data in Text.
- If a processing instruction is encountered (
<?...?>), Text holds the text of the processing instruction. Please note that the
<?xml ...?>instruction is handled internally.
The Options list controls the conversion process. Currently defined options are below. Other options are passed to sgml_parse/2.
- Reference to a DTD object. If specified, the
<!DOCTYPE ...>declaration is ignored and the document is parsed and validated against the provided DTD. If provided as a variable, the created DTD is returned. See section 3.5.
- Specify the parsing dialect. Supported are
xmlns. See the option
dialectof set_sgml_parser/2 for details.
- Define whether SHORTTAG abbreviation is accepted. The default is true
for SGML mode and false for the XML modes. Without SHORTTAG, a
is accepted with warning as part of an unquoted attribute-value, though
/>still closes the element-tag in XML mode. It may be set to false for parsing HTML documents to allow for unquoted URLs containing
- Sets the‘space-handling-mode' for the initial environment. This
mode is inherited by the other environments, which can override the
inherited value using the XML reserved attribute
xml:space. See section 3.2.
- Determines how attributes of type
NUMBERSare handled. If
token(default) they are passed as an atom. If
integerthe parser attempts to convert the value to an integer. If successful, the attribute is passed as a Prolog integer. Otherwise it is still passed as an atom. Note that SGML defines a numeric attribute to be a sequence of digits. The
sign is not allowed and
1is different from
01. For this reason the default is to handle numeric attributes as tokens. If conversion to integer is enabled, negative values are silently accepted.
- Treat attribute values as case sensitive. The default is
truefor XML and
falsefor SGML and HTML dialects.
- Treat attribute values as case insensitive but do not alter their case.
The default is
false. Setting this option sets the
case_sensitive_attributesto the same value. This option was added to support HTML quasi quotations and most likely has little value in other contexts.
- Define whether SYSTEM entities are expanded. The default is
- Determines how default and fixed values from the DTD are used. By
default, defaults are included in the output if they do not appear in
the source. If
false, only the attributes occurring in the source are emitted.
- entity(+Name, +Value)
- Defines (overwrites) an entity definition. At the moment, only
CDATAentities can be specified with this construct. Multiple entity options are allowed.
- Sets the name of the file on which errors are reported. Sets the linenumber to 1.
- Sets the starting line-number for reporting errors.
- Sets the maximum buffer size in bytes available for input data and CDATA
output. If this limit is reached a resource error is raised. Using
max_memory(0)(the default) means no resource limit will be enforced.
- Specify the representation of cdata elements. Supported are
string. The choice is not obvious. Strings are allocated on the Prolog stacks and subject to normal stack garbage collection. They are quicker to create and avoid memory fragmentation. But, multiple copies of the same string are stored multiple times, while the text is shared if atoms are used. Strings are also useful for security sensitive information as they are invisible to other threads and cannot be enumerated using, e.g., current_atom/1. Finally, using strings allows for resource usage limits using the global stack limit (see set_prolog_stack/2).
- Specify the representation of attribute values. Supported are
string. See above for the advantages and disadvantages of using strings.
true, xmlns namespaces with prefixes are returned as
ns(Prefix, URI)terms. If
false(default), the prefix is ignored and the xmlns namespace is returned as just the URI.
SGML2PL has four modes for handling white-space. The initial mode can
be switched using the
space(SpaceMode) option to
In XML mode, the mode is further controlled by the
attribute, which may be specified both in the DTD and in the document.
The defined modes are:
- In SGML, newlines at the start and end of an element are removed.2In addition, newlines at the end of lines containing only markup should be deleted. This is not yet implemented. This is the default mode for the SGML dialect.
- White space is passed literally to the application. This mode leaves all white space handling to the application. This is the default mode for the XML dialect.
- In addition to
sgmlspace-mode, all consequtive white-space is reduced to a single space-character. This mode canonicalises all white space.
- In addition to
default, all leading and trailing white-space is removed from
CDATAobjects. If, as a result, the
CDATAbecomes empty, nothing is passed to the application. This mode is especially handy for processing‘data-oriented' documents, such as RDF. It is not suitable for normal text documents. Consider the HTML fragment below. When processed in this mode, the spaces between the three modified words are lost. This mode is not part of any standard; XML 1.0 allows only
Consider adjacent <b>bold</b> <ul>and</ul> <it>italic</it> words.
The parser can operate in two modes:
sgml mode and
mode, as defined by the
dialect(Dialect) option. Regardless
of this option, if the first line of the document reads as below, the
parser is switched automatically into XML mode.
<?xml ... ?>
Currently switching to XML mode implies:
- XML empty elements
<element [attribute...] />is recognised as an empty element.
- Predefined entities
The following entitities are predefined:
- Case sensitivity
In XML mode, names are treated case-sensitive, except for the DTD reserved names (i.e.
- Character classes
In XML mode, underscores (
_) and colon (
:) are allowed in names.
- White-space handling
White space mode is set to
preserve. In addition to setting white-space handling at the toplevel the XML reserved attribute
xml:spaceis honoured. It may appear both in the document and the DTD. The
removeextension is honoured as
xml:spacevalue. For example, the DTD statement below ensures that the
preelement preserves space, regardless of the default processing mode.
<!ATTLIST pre xml:space nmtoken #fixed preserve>
Using the dialect
xmlns, the parser will
interpret XML namespaces. In this case, the names of elements are
returned as a term of the format
If an identifier has no namespace and there is no default namespace it is returned as a simple atom. If an identifier has a namespace but this namespace is undeclared, the namespace name rather than the related URL is returned.
Attributes declaring namespaces (
are reported as if
xmlns were not a defined resource.
In many cases, getting attribute-names as url:name is not desirable. Such terms are hard to unify and sometimes multiple URLs may be mapped to the same identifier. This may happen due to poor version management, poor standardisation or because the the application doesn't care too much about versions. This package defines two call-backs that can be set using set_sgml_parser/2 to deal with this problem.
xmlns is called as XML namespaces are
noticed. It can be used to extend a canonical mapping for later use by
urlns call-back. The following illustrates this
behaviour. Any namespace containing
rdf-syntax in its URL
or that is used as
rdf namespace is canonicalised to
implies that any attribute and element name from the RDF namespace
:- dynamic xmlns/3. on_xmlns(rdf, URL, _Parser) :- !, asserta(xmlns(URL, rdf, _)). on_xmlns(_, URL, _Parser) :- sub_atom(URL, _, _, _, 'rdf-syntax'), !, asserta(xmlns(URL, rdf, _)). load_rdf_xml(File, Term) :- load_structure(File, Term, [ dialect(xmlns), call(xmlns, on_xmlns), call(urlns, xmlns) ]).
The library provides iri_xml_namespace/3 to break down an IRI into its namespace and localname:
- [det]iri_xml_namespace(+IRI, -Namespace, -Localname)
- Split an IRI (Unicode URI) into its Namespace (an IRI) and
Localname (a Unicode XML name, see xml_name/2).
Localname is defined as the longest last part of the IRI that
satisfies the syntax of an XML name. With IRI schemas that are designed
to work with XML namespaces, this will typically break the IRI on the
. Note however that this can produce unexpected results. E.g., in the example below, one might expect the namespace to be http://example.com/images\#, but an XML name cannot start with a digit.
?- iri_xml_namespace('http://example.com/images#12345', NS, L). NS = 'http://example.com/images#12345', L = ''.
As we see from the example above, the Localname can be the empty atom. Similarly, Namespace can be the empty atom if IRI is an XML name. Applications will often have to check for either or both these conditions. We decided against failing in these conditions because the application typically wants to know which of the two conditions (empty namespace or empty localname) holds. This predicate is often used for generating RDF/XML from an RDF graph.
- [det]iri_xml_namespace(+IRI, -Namespace)
- Same as iri_xml_namespace/3, but avoids creating an atom for the Localname.
The DTD (Document Type Definition) is a separate entity in sgml2pl, that can be created, freed, defined and inspected. Like the parser itself, it is filled by opening it as a Prolog output stream and sending data to it. This section summarises the predicates for handling the DTD.
- new_dtd(+DocType, -DTD)
- Creates an empty DTD for the named DocType. The returned DTD-reference is an opaque term that can be used in the other predicates of this package.
- Deallocate all resources associated to the DTD. Further use of DTD is invalid.
- load_dtd(+DTD, +File)
- Define the DTD by loading the SGML-DTD file File. Same as load_dtd/3 with empty option list.
- load_dtd(+DTD, +File, +Options)
- Define the DTD by loading File. Defined options are the
dialectoption from open_dtd/3 and the
encodingoption from open/4. Notably the
dialectoption must match the dialect used for subsequent parsing using this DTD.
- open_dtd(+DTD, +Options, -OutStream)
- Open a DTD as an output stream. See load_dtd/2
for an example. Defined options are:
- Define the DTD dialect. Default is
xmlnsprocesses the DTD case-sensitive.
- dtd(+DocType, -DTD)
- Find the DTD representing the indicated doctype. This predicate
uses a cache of DTD objects. If a doctype has no associated dtd, it
searches for a file using the file search path
dtdusing the call:
..., absolute_file_name(dtd(Type), [ extensions([dtd]), access(read) ], DtdFile), ...
Note that DTD objects may be modified while processing errornous documents. For example, loading an SGML document starting with
<?xml ...?>switches the DTD to XML mode and encountering unknown elements adds these elements to the DTD object. Re-using a DTD object to parse multiple documents should be restricted to situations where the documents processed are known to be error-free.
htmlis handled separately. The Prolog flag
html_dialectspecifies the default html dialect, which is either
html5(default).3Note that HTML5 has no DTD. The loaded DTD is an informal DTD that includes most of the HTML5 extensions (http://www.cs.tut.fi/~jkorpela/html5-dtd.html). In addition, the parser sets the
dialectflag of the DTD object. This is used by the parser to accept HTML extensions. Next, the corresponding DTD is loaded.
- dtd_property(+DTD, ?Property)
- This predicate is used to examine the content of a DTD. Property is one
- An atom representing the document-type defined by this DTD.
- A list of atoms representing the names of the elements in this DTD.
- element(Name, Omit, Content)
- The DTD contains an element with the given name. Omit is a
term of the format
omit(OmitOpen, OmitClose), where both arguments are booleans (
falserepresenting whether the open- or close-tag may be omitted. Content is the content-model of the element represented as a Prolog term. This term takes the following form:
- The element has no content.
- The element contains non-parsed character data. All data up to the matching end-tag is included in the data (declared content).
cdata, but entity-references are expanded.
- The element may contain any number of any element from the DTD in any order.
- The element contains parsed character data .
- n element with this name.
- 0 or more appearances.
- 0 or one appearance.
- 1 or more appearances.
- SubModel1 followed by SubModel2.
- &(SubModel1, SubModel2)
- SubModel1 and SubModel2 in any order.
- SubModel1 or SubModel2.
- attributes(Element, ListOfAttributes)
- ListOfAttributes is a list of atoms representing the attributes of the element Element.
- attribute(Element, Attribute, Type, Default)
- Query an element. Type is one of
nutoken. For DTD types that allow for a list, the notation
list(Type)is used. Finally, the DTD construct
(a|b|...)is mapped to the term
Default describes the sgml default. It is one
implied. If a real default is present, it is one of
- ListOfEntities is a list of atoms representing the names of the defined entities.
- entity(Name, Value)
- Name is the name of an entity with given value. Value is one
- If the value is atomic, it represents the literal value of the entity.
- Url is the URL of the system external entity.
- public(Id, Url)
- For external public entities, Id is the identifier. If an URL is provided this is returned in Url. Otherwise this argument is unbound.
- Returns a list holding the names of all
- notation(Name, Decl)
- Unify Decl with a list if
As this parser allows for processing partial documents and process the DTD separately, the DOCTYPE declaration plays a special role.
If a document has no DOCTYPE declaraction, the parser returns a list holding all elements and CDATA found. If the document has a DOCTYPE declaraction, the parser will open the element defined in the DOCTYPE as soon as the first real data is encountered.
Some documents have no DTD. One of the neat facilities of this
library is that it builds a DTD while parsing a document with an
implicit DTD. The resulting DTD contains all elements
encountered in the document. For each element the content model is a
disjunction of elements and possibly
#PCDATA that can be
repeated. Thus, if we found element
y and CDATA in element
x, the model is:
<!ELEMENT x - - (y|#PCDATA)*>
Any encountered attribute is added to the attribute list with the
CDATA and default
The example below extracts the elements used in an unknown XML document.
elements_in_xml_document(File, Elements) :- load_structure(File, _, [ dialect(xml), dtd(DTD) ]), dtd_property(DTD, elements(Elements)), free_dtd(DTD).
- new_sgml_parser(-Parser, +Options)
- Creates a new parser. A parser can be used one or multiple times for
parsing documents or parts thereof. It may be bound to a DTD or the DTD
may be left implicit, in which case it is created from the document
prologue or parsing is performed without a DTD. Options:
- If specified with an initialised DTD, this DTD is used for parsing the document, regardless of the document prologue. If specified using as a variable, a reference to the created DTD is returned. This DTD may be created from the document prologue or build implicitely from the document's content.
- Destroy all resources related to the parser. This does not destroy the
DTD if the parser was created using the
- set_sgml_parser(+Parser, +Option)
- Sets attributes to the parser. Currently defined attributes:
- Sets the file for reporting errors and warnings. Sets the line to 1.
- Sets the current line. Useful if the stream is not at the start of the (file) object for generating proper line-numbers.
- Sets notion of the current column in the source line.
- Sets the current character location. See also the
- Set source location from a stream position term as obtained using
- Set the markup dialect. Known dialects:
- The default dialect is to process as SGML. This implies markup is case-insensitive and standard SGML abbreviation is allowed (abreviated attributes and omitted tags).
- This is the same as
sgml, but implies
shorttag(false)and accepts XML empty element declarations (e.g.,
- In addition to
html, accept attributes named
data-without warning. This value initialises the charset to UTF-8.
- These document types are processed as
xhtml5accepts attributes named
- This dialect is selected automatically if the processing instruction
<?xml ...>is encountered. See section 3.3 for details.
- Process file as XML file with namespace support. See section
3.3.1 for details. See also the
- Set the default namespace of the outer environment. This option is provided to process partial XML content with proper namespace resolution.
- xmlns(+NS, +URI)
- Specify a namespace for the outer environment. This option is provided to process partial XML content with proper namespace resolution.
- How to handle unqualified attribute (i.e. without an explicit namespace)
in XML namespace (
xmlns) mode. Default and standard compliant is not to qualify such elements. If
true, such attributes are qualified with the namespace of the element they appear in. This option is for backward compatibility as this is the behaviour of older versions. In addition, the namespace document suggests unqualified attributes are often interpreted in the namespace of their element.
- Define the initial handling of white-space in PCDATA. This attribute is described in section 3.2.
token(default), attributes of type number are passed as a Prolog atom. If
integer, such attributes are translated into Prolog integers. If the conversion fails (e.g. due to overflow) a warning is issued and the value is passed as an atom.
- Set the initial encoding. The default initial encoding for XML documents
is UTF-8 and for SGML documents ISO-8859-1. XML documents may change the
encoding using the
encoding=attribute in the header. Explicit use of this option is only required to parse non-conforming documents. Currently accepted values are
- Defines the toplevel element expected. If a
<!DOCTYPEdeclaration has been parsed, the default is the defined doctype. The parser can be instructed to accept the first element encountered as the toplevel using
doctype(_). This feature is especially useful when parsing part of a document (see the
parseoption to sgml_parse/2.
- get_sgml_parser(+Parser, -Option)
- Retrieve infomation on the current status of the parser. Notably useful
if the parser is used in the call-back mode. Currently defined options:
- Current file-name. Note that this may be different from the provided file if an external entity is being loaded.
- Line-offset from where the parser started its processing in the file-object.
- Offset from where the parser started its processing in the file-object. See section 6.
- charpos(-Start, -End)
- Character offsets of the start and end of the source processed causing the current call-back. Used in PceEmacs to for colouring text in SGML and XML modes.
- Prolog stream being processed. May be used in the
on_begin, etc. callbacks from sgml_parse/2.
- Return the current dialect used by the parser (
- The event class can be requested in call-back events. It
denotes the cause of the event, providing useful information for syntax
highlighting. Defined values are:
- The code generating this event is explicitely present in the document.
- The current event is caused by the insertion of an omitted tag. This may be a normal event in SGML mode or an error in XML mode.
- The current event (
end) is caused by an element written down using the shorttag notation (
- The current event is caused by the expansion of a shortref. This allows for highlighting shortref strings in the source-text.
- Return the defined document-type (= toplevel element). See also set_sgml_parser/2.
- Return the currently used DTD. See dtd_property/2 for obtaining information on the DTD such as element and attribute properties.
- Returns the stack of currently open elements as a list. The head of this list is the current element. This can be used to determine the context of, for example, CDATA events in call-back mode. The elements are passed as atoms. Currently no access to the attributes is provided.
- Determines which elements may be inserted at the current location. This
information is returned as a list of element-names. If character data is
allowed in the current location,
#pcdatais part of Elements. If no element is open, the doctype is returned.
This option is intended to support syntax-sensitive editors. Such an editor should load the DTD, find an appropriate starting point and then feed all data between the starting point and the caret into the parser. Next it can use this option to determine the elements allowed at this point. Below is a code fragment illustrating this use given a parser with loaded DTD, an input stream and a start-location.
..., seek(In, Start, bof, _), set_sgml_parser(Parser, charpos(Start)), set_sgml_parser(Parser, doctype(_)), Len is Caret - Start, sgml_parse(Parser, [ source(In), content_length(Len), parse(input) % do not complete document ]), get_sgml_parser(Parser, allowed(Allowed)), ...
- sgml_parse(+Parser, +Options)
- Parse an XML file. The parser can operate in two input and two output
modes. Output is either a structured term as described with
or call-backs on predefined events. The first is especially suitable for
manipulating not-too-large documents, while the latter provides a
primitive means for handling very large documents.
Input is a stream. A full description of the option-list is below.
- A variable that will be unified with a list describing the content of the document (see load_structure/2).
- An input stream that is read. This option must be given.
- Stop parsing after Characters. This option is useful to parse input embedded in envelopes, such as the HTTP protocol.
- Specify the representation of cdata elements. Supported are
string. See load_structure/3 for details.
- Defines how much of the input is parsed. This option is used to parse
only parts of a file.
- Default. Parse everything upto the end of the input.
- The parser stops after reading the first element. Using
source(Stream), this implies reading is stopped as soon as the element is complete, and another call may be issued on the same stream to read the next element.
- The value
elementbut assumes the element has already been opened. It may be used in a call-back from
call(to parse individual elements after validating their headers.
- This may be used to stop the parser after reading the first declaration.
This is especially useful to parse only the
- This option is intended to be used in conjunction with the
allowed(Elements)option of get_sgml_parser/2. It disables the parser's default to complete the parse-tree by closing all open elements.
- Set the maximum number of errors. If this number is exceeded further
writes to the stream will yield an I/O error exception. Printing of
errors is suppressed after reaching this value. The default is 50. Using
max_errors(-1)makes the parser continue, no matter how many errors it encounters.
error(limit_exceeded(max_errors, Max), _)
- Defines how syntax errors are handled.
- Error handling if an XML namespace is not defined. Default generates an
quiet, the error is suppressed. Can be used together with
call(urlns, Closure)to provide external expansion of namespaces. See also section 3.3.1.
- call(+Event, :PredicateName)
- Issue call-backs on the specified events. PredicateName is
the name of the predicate to call on this event, possibly prefixed with
a module identifier. If the handler throws an exception, parsing is
stopped and sgml_parse/2
re-throws the exception. The defined events are:
- An open-tag has been parsed. The named handler is called with three
Handler(+Tag, +Attributes, +Parser).
- A close-tag has been parsed. The named handler is called with two
- CDATA has been parsed. The named handler is called with two arguments:
Handler(+CDATA, +Parser), where CDATA is an atom representing the data.
- A processing instruction has been parsed. The named handler is called
with two arguments:
Handler(+Text, +Parser), where Text is the text of the processing instruction.
- A declaration (
<!...>) has been read. The named handler is called with two arguments:
Handler(+Text, +Parser), where Text is the text of the declaration with comments removed.
This option is expecially useful for highlighting declarations and comments in editor support, where the location of the declaration is extracted using get_sgml_parser/2.
- An error has been encountered. the named handler is called with three
Handler(+Severity, +Message, +Parser), where Severity is one of
errorand Message is an atom representing the diagnostic message. The location of the error can be determined using get_sgml_parser/2
If this option is present, errors and warnings are not reported using print_message/3
- When parsing an in
xmlnsmode, a new namespace declaraction is pushed on the environment. The named handler is called with three arguments:
Handler(+NameSpace, +URL, +Parser). See section 3.3.1 for details.
- When parsing an in
xmlnsmode, this predicate can be used to map a url into either a canonical URL for this namespace or another internal identifier. See section 3.3.1 for details.
In some cases, part of a document needs to be parsed. One option is
to use load_structure/2
or one of its variations and extract the desired elements from the
returned structure. This is a clean solution, especially on small and
medium-sized documents. It however is unsuitable for parsing really big
documents. Such documents can only be handled with the call-back output
interface realised by the
call(Event, Action) option of sgml_parse/2.
Event-driven processing is not very natural in Prolog.
The SGML2PL library allows for a mixed approach. Consider the case
where we want to process all descriptions from RDF elements in a
document. The code below calls
on each element that is directly inside an RDF element.
:- dynamic in_rdf/0. load_rdf(File) :- retractall(in_rdf), open(File, read, In), new_sgml_parser(Parser, ), set_sgml_parser(Parser, file(File)), set_sgml_parser(Parser, dialect(xml)), sgml_parse(Parser, [ source(In), call(begin, on_begin), call(end, on_end) ]), close(In). on_end('RDF', _) :- retractall(in_rdf). on_begin('RDF', _, _) :- assert(in_rdf). on_begin(Tag, Attr, Parser) :- in_rdf, !, sgml_parse(Parser, [ document(Content), parse(content) ]), process_rdf_description(element(Tag, Attr, Content)).
- True if Term is an SGML/XML term as produced by one of the above predicates and acceptable by xml_write/3 and friends.