IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
Even with flush set to true, xmlCharEncInput didn't guarantee to decode
all data. This complicated the push parser.
Remove the flush flag and always decode all available data.
Also fix ICU code where the flush flag has a different meaning. Always
set flush to false and retry even with empty input buffers.
Don't create a copy of the whole input buffer. Read the data chunk by
chunk to save memory.
Historically, it was probably envisioned to read data from memory
without additional copying. This doesn't work reliably with the current
design of the XML parser which requires a terminating null byte at the
end of input buffers. This lead to xmlReadMemory interfaces, which
expect pointer and size arguments, being changed to make a
zero-terminated copy of the input buffer. Interfaces based on
xmlReadDoc, which actually expect a zero-terminated string and
would make zero-copy operation work, were then simplified to rely on
xmlReadMemoryi, resulting in an unnecessary copy.
To avoid copying (possibly gigabytes) of memory temporarily, we now
stream in-memory input just like content read from files in a
chunk-by-chunk fashion (using a somewhat outdated INPUT_CHUNK size of
250 bytes). As a side effect, we also avoid another copy of the whole
input when handling non-UTF-8 data which was made possible by some
earlier commits.
Interfaces expecting zero-terminated strings now make use of strnlen
which unfortunately isn't part of the standard C library and only
mandated since POSIX 2008.
Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set
when xmlSwitchEncoding is called. The parser can use the flag to
reliably detect whether an encoding was already set via user override,
BOM or other auto-detection. In this case, the encoding declaration
won't be used to switch the encoding.
Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding
and ctxt->input->buf->encoder was used.
Introduce private helper functions to switch encodings used by both the
XML and HTML parser:
- xmlDetectEncoding which skips over the BOM, allowing to remove the
BOM checks from other encoding functions.
- xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns
about encoding mismatches.
If users override the encoding, store the declared instead of the actual
encoding in xmlDoc. In this case, the actual encoding is known and the
raw value from the doc is more useful.
Also use the input flags to store the ISO-8859-1 fallback state.
Restrict the fallback to cases where no encoding was specified. (The
fallback is only useful in recovery mode and these days broken UTF-8 is
probably more likely than ISO-8859-1, so it might eventually be removed
completely.)
The 'charset' member of xmlParserCtxt is now unused. The 'encoding'
member of xmlParserInput is now unused.
The 'standalone' member of xmlParserInput is renamed to 'flags'.
A new parser state XML_PARSER_XML_DECL is added for the push parser.
Chromium calls xmlSwitchEncoding from the start document handler and
relies on this function to skip the BOM. Commit 98840d40 changed the
behavior when switching to UTF-16 since inspecting the input buffer at
this point is fragile.
Revert part of the commit to also skip a potential (decoded UTF-8) BOM
when switching to UTF-16. Make sure that we do this only at the start of
an input stream to avoid U-FEFF characters being lost.
BOM handling should ultimately be moved to the parsing code to avoid
such bugs.
See https://bugs.chromium.org/p/chromium/issues/detail?id=1451026
Fix short-lived regression from previous commit.
It might be safer to make xmlBufSetInputBaseCur use the original buffer
even in case of errors.
Found by OSS-Fuzz.
Make sure that xmlCharEncInput, xmlParserInputBufferPush and
xmlParserInputBufferGrow set the correct error code in the
xmlParserInputBuffer. Handle errors when calling these functions.
Revert another change from commit 98840d40.
Decode the whole buffer when reading from memory and switching to the
initial encoding. Add some comments about potential improvements.
Revert some changes from commit 98840d40.
WebKit/Chromium can actually switch from ISO-8859-1 to UTF-16 in the
middle of parsing. This is a bad idea, but we have to keep supporting
this use case.
To detect EBCDIC code pages, we used to switch the encoding twice and
had to be very careful not to decode data after the XML declaration
before the second switch. This relied on a hard-coded expected size of
the XML declaration and was complicated and unreliable.
Now we convert the first 200 bytes to EBCDIC-US and parse the encoding
declaration manually.
Don't try to grow the input buffer in xmlParserShrink. This makes sure
that no memory allocations are made and the function always succeeds.
Remove unnecessary invocations of SHRINK. Invoke SHRINK at the end of
DTD parsing loops.
Shrink before growing.
This commit implements robust detection of entity amplification attacks,
better known as the "billion laughs" attack.
We now limit the size of the document after substitution of entities to
10 times the size before expansion. This guarantees linear behavior by
definition. There already was a similar check before, but the accounting
of "sizeentities" (size of external entities) and "sizeentcopy" (size of
all copies created by entity references) wasn't accurate.
We also need saturation arithmetic since we're historically limited to
"unsigned long" which is 32-bit on many platforms.
A maximum of 10 MB of substitutions is always allowed. This should make
use cases like DITA work which have caused problems in the past.
The old checks based on the number of entities were removed. This is
accounted for by adding a fixed cost to each entity reference.
Entity amplification checks are now enabled even if XML_PARSE_HUGE is
set. This option is mainly used to allow larger text nodes. Most users
were unaware that it also disabled entity expansion checks.
Some of the limits might be adjusted later. If this change turns out to
affect legitimate use cases, we can add a separate parser option to
disable the checks.
Fixes#294.
Fixes#345.
In some cases, for example when using encoders, the read callback was
set to NULL, in other cases it was set to xmlInputReadCallbackNop.
xmlGROW only tested for xmlInputReadCallbackNop, resulting in errors
when parsing large encoded content from memory.
Always use a NULL callback for memory buffers to avoid ambiguities.
Fixes#262.
Remove explicit integer casts as final operation
- in assignments
- when passing arguments
- when returning values
Remove casts
- to the same type
- from certain range-bound values
The main motivation is that these explicit casts don't change the result
of operations and only render UBSan's implicit-conversion checks
useless. Removing these casts allows UBSan to detect cases where
truncation or sign-changes occur unexpectedly.
Document some explicit casts as truncating and add a few missing ones.
Private functions were previously declared
- in header files in the root directory
- in public headers guarded with IN_LIBXML
- in libxml.h
- redundantly in source files that used them.
Consolidate all private header files in include/private.
- Remove xmlSwitchToEncodingInt which was basically just a wrapper
around xmlSwitchInputEncodingInt.
- Simplify xmlSwitchEncoding.
- Improve error handling in xmlSwitchInputEncodingInt.
- Deprecate xmlSwitchInputEncoding.
From what I can tell, some really early Cygwin versions from around
1998-2000 used to erroneously define _WIN32. This was eventually fixed,
but these days, the `defined(_WIN32) && !defined(__CYGWIN__)` idiom is
unnecessary.
Now, we only check for __CYGWIN__ in xmlexports.h when deciding whether
to use __declspec.
Use a bitmask instead of magic values to
- keep track whether the validation context is part of a parser context
- keep track whether xmlValidateDtdFinal was called
This allows to add addtional flags later.
Note that this deliberately changes the name of a public struct member,
assuming that this was always private data never to be used by client
code.
Under certain circumstances, the HTML parser would try to guess and
switch input encodings multiple times, leading to slow processing of
documents with encoding errors. The repeated scanning of the input
buffer when guessing encodings could even lead to quadratic behavior.
The code htmlCurrentChar probably assumed that if there's an encoding
handler, it is guaranteed to produce valid UTF-8. This holds true in
general, but if the detected encoding was "UTF-8", the UTF8ToUTF8
encoding handler simply invoked memcpy without checking for invalid
UTF-8. This still must be fixed, preferably by not using this handler
at all.
Also leave a note that switching encodings twice seems impossible to
implement correctly. Add a check when handling UTF-8 encoding errors
in htmlCurrentChar to avoid this situation, even if encoders produce
invalid UTF-8.
Found by OSS-Fuzz.
If libz or liblzma are detected with pkg-config, AC_CHECK_HEADERS must
not be run because the correct CPPFLAGS aren't set. It is actually not
required have separate checks for LIBXML_ZLIB_ENABLED and HAVE_ZLIB_H.
Only check for LIBXML_ZLIB_ENABLED and remove HAVE_ZLIB_H macro.
Fixes bug 764657, bug 787041.
Make sure that all parameters and return values of hash callback
functions exactly match the callback function type. This is required
to pass clang's Control Flow Integrity checks and to allow compilation
to asm.js with Emscripten.
Fixes bug 784861.
Don't include windows.h and wsockcompat.h from config.h but only when
needed.
Don't define _WINSOCKAPI_ manually. This was apparently done to stop
windows.h from including winsock.h which is a problem if winsock2.h
wasn't included first. But on MinGW, this causes compiler warnings.
Define WIN32_LEAN_AND_MEAN instead which has the same effect.
Always use the compiler-defined _WIN32 macro instead of WIN32.
This reverts commit 79c8a6b which caused a serious regression in
streaming mode.
Also reverts part of commit 52ceced "Fix infinite loops with push
parser in recovery mode".
Fixes bug 786554.
It's enough if xmlPopInput is called from xmlSkipBlankChars. Since the
replacement text of a parameter entity is surrounded with space
characters, that's the only place where the replacement can end in a
well-formed document.
This is also required to get rid of the "blanks wrapper" hack.
There are only two places where parameter entity references must be
handled. For the internal subset in xmlParseInternalSubset. For the
external subset or content from other external PEs in xmlSkipBlankChars.
Make sure that xmlSkipBlankChars skips over sequences of PEs and
whitespace. Rely on xmlSkipBlankChars instead of calling
xmlParsePEReference directly when in the external subset or a
conditional section.
xmlParserHandlePEReference is unused now.
Before, truncated UTF-8 sequences at the end of a file were treated as
EOF. Create an error message containing the offending bytes.
xmlStringCurrentChar would also print characters from the input stream,
not the string it's working on.
Call xmlBufResetInput before bailing out if switching the encoding
fails. Otherwise, the input pointers are left in an invalid state.
This would typically lead to an internal error in xmlGROW but could also
cause other unforeseen problems.
For https://bugzilla.gnome.org/show_bug.cgi?id=758606
* parserInternals.c:
(xmlNextChar): Add an test to catch other issues on ctxt->input
corruption proactively.
For non-UTF-8 charsets, xmlNextChar() failed to check for the end
of the input buffer and would continuing reading. Fix this by
pulling out the check for the end of the input buffer into common
code, and return if we reach the end of the input buffer
prematurely.
* result/HTML/758606.html: Added.
* result/HTML/758606.html.err: Added.
* result/HTML/758606.html.sax: Added.
* result/HTML/758606_2.html: Added.
* result/HTML/758606_2.html.err: Added.
* result/HTML/758606_2.html.sax: Added.
* test/HTML/758606.html: Added test case.
* test/HTML/758606_2.html: Added test case.
For https://bugzilla.gnome.org/show_bug.cgi?id=756527
and was also raised by Chromium team in the past
When we hit a convwersion failure when switching encoding
it is bestter to stop parsing there, this was treated as a
fatal error but the parser was continuing to process to extract
more errors, unfortunately that makes little sense as the data
is obviously corrupt and can potentially lead to unexpected behaviour.
If entities expansion in the XML parser is asked for,
it is possble to craft relatively small input document leading
to excessive on-the-fly content generation.
This patch accounts for those replacement and stop parsing
after a given threshold. it can be bypassed as usual with the
HUGE parser option.
https://bugzilla.gnome.org/show_bug.cgi?id=692915
the new set of converting functions tried to limit the encoding
conversion of the raw buffer to the consumption one to work in
a more progressive fashion. Unfortunately this was bad for
performances and led to errors on progressive parsing when
a very large chunk was close to the end of the document. Fix
the new internal function and switch back to the old way of
converting. Fix another bug in the process.
Those can be overrided by the XML_PARSE_HUGE option, they
are just default limits for Name lenght, dictionary size limits
and maximum amount of parser lookup.
* include/libxml/parserInternals.h: define the limits
* include/libxml/xmlerror.h: add a new error
* parser.c parserInternals.c: implements the new limits
This was scattered in a number of modules, xmlParserInputPtr
have usually their base, cur and end pointer set from an
xmlBuf used as input.
* buf.c buf.h: add a new function implementing this setup
* parser.c HTMLparser.c catalog.c parserInternals.c xmlreader.c
use the new function instead of digging into the buffer in
all those modules
The main changes are when the internal of the buffers structure
were adressed directly, we now use routines coming from buf.h
The routine xmlParserInputRead() which wasn't used anywhere is
deprecated too.
For https://bugzilla.gnome.org/show_bug.cgi?id=643148
Reported by Bill Clarke <llib@computer.org>, it used a global variable
as a counter for the input id and this was not thread safe. To avoid the
race without adding unneeded locking in the parser path, move the id to
the parser context instead.
* HTMLparser.c: new htmlParseElementInternal non recursive, with
htmlParseContentInternal and new function to handle node info
and element end.
* include/libxml/parser.h: add new stack for element info in parser
context
* parserInternals.c: fee element info stack
* encoding.c parser.c parserInternals.c: when we autodetect an encoding
but it's actually not completely compatible with the one declared
great care must be taken to not convert more than just the first line.
Led to some refactoring, more private functions and a bit of cleanup.
* parser.c: avoid a warning on 64bits introduced earlier
* parserInternals.c: make more checking on the UTF-8 input
Daniel
svn path=/trunk/; revision=3676
* HTMLparser.c configure.in parserInternals.c runsuite.c runtest.c
testapi.c xmlschemas.c xmlschemastypes.c xmlstring.c: fixed a number
of warnings shown by HP-UX compiler and reported by Rick Jones
Daniel
* encoding.c parserInternals.c: avoid passing a char[] as snprintf
first argument.
* threads.c include/libxml/threads.h: implemented xmlIsThreadsEnabled()
based on Andrew W. Nosenko idea.
* doc/* elfgcchack.h: regenerated the API
Daniel
* parserInternals.c: fix bug raised by zamez on IRC
* testapi.c: regenerated, seems to pop-up leaks in new tree functions
* tree.c: added comments missing.
* doc/*: regenerated
Daniel
* doc/apibuild.py doc/elfgcchack.xsl: revamped the elfgcchack.h
format to cope with gcc4 change of aliasing allowed scopes, had
to add extra informations to doc/libxml2-api.xml to separate
the header from the c module source.
* *.c: updated all c library files to add a #define bottom_xxx
and reimport elfgcchack.h thereafter, and a bit of cleanups.
* doc//* testapi.c: regenerated when rebuilding the API
Daniel
* parserInternals.c: fixed to skip (if necessary) the BOM for
encoding 'utf-16'. Completes the fix for bug #152286.
* tree.c, parser.c: minor warning cleanup, no change to logic
* gentest.py testapi.c: better handling of conditional features
* HTMLparser.c SAX2.c parserInternals.c xmlwriter.c: more testing
on parser contexts closed leaks, error messages
Daniel
* gentest.py testapi.c: autogenerate a minimal NULL value sequence
for unknown pointer types
* HTMLparser.c SAX2.c chvalid.c encoding.c entities.c parser.c
parserInternals.c relaxng.c valid.c xmlIO.c xmlreader.c
xmlsave.c xmlschemas.c xmlschemastypes.c xmlstring.c xpath.c
xpointer.c: This uncovered an impressive amount of entry points
not checking for NULL pointers when they ought to, closing all
the open gaps.
Daniel
* xmlmemory.c include/libxml/xmlmemory.h: adding xmlMemBlocks()
* Makefile.am gentest.py testapi.c: work on generator of an
automatic API regression test tool.
* SAX2.c nanoftp.c parser.c parserInternals.c tree.c xmlIO.c
xmlstring.c: various API hardeing changes as a result of running
teh first set of automatic API regression tests.
* test/slashdot16.xml: apparently missing from CVS, commited it
Daniel
* python/libxml.c: register xmlSchemaSetValidErrors, patch from
Brent Hendricks in the mailing-list
* include/libxml/valid.h HTMLparser.c SAX2.c valid.c
parserInternals.c: fix#156626 and more generally how to find out
if a validation contect is part of a parsing context or not. This
can probably be improved to make 100% sure that vctxt->userData
is the parser context too. It's a bit hairy because we can't
change the xmlValidCtxt structure without breaking the ABI since
this change xmlParserCtxt information indexes.
Daniel
* xmlIO.c: small typo pointed out by Mike Hommey
* doc/xmllint.xml, xmllint.html, xmllint.1: slightly improved
the --c14n description, c.f. #144675 .
* nanohttp.c nanoftp.c: applied a first simple patch from
Mike Hommey for $no_proxy, c.f. #133470
* parserInternals.c include/libxml/parserInternals.h
include/libxml/xmlerror.h: cleanup to avoid 'error' identifier
in includes #
* parser.c SAX2.c debugXML.c include/libxml/parser.h:
first version of the inplementation of parsing within
the context of a node in the tree #142359, new function
xmlParseInNodeContext(), added support at the xmllint --shell
level as the "set" function
* test/scripts/set* result/scripts/* Makefile.am: extended
the script based regression tests to instrument the new function.
Daniel
* parserInternals.c xmlIO.c encoding.c include/libxml/parser.h
include/libxml/xmlIO.h: added xmlByteConsumed() interface
* doc/*: updated the benchmark rebuilt the docs
* python/tests/Makefile.am python/tests/indexes.py: added a
specific regression test for xmlByteConsumed()
* include/libxml/encoding.h rngparser.c tree.c: small cleanups
Daniel
* xinclude.c: remove the warning on the 2001 namespace
* parser.c parserInternals.c xpath.c: remove some warnings
when compiling with MSVC6
* nanohttp.c: applied a patch when using _WINSOCKAPI_
Daniel
* python/libxml.c: previous fix for #124044 was broken, correct
fix provided.
* HTMLparser.c parser.c parserInternals.c xmlIO.c: fix xmlStopParser()
and the error handlers to address #125877
Daniel
* libxml.h parserInternals.c xmlIO.c: make sure we report errors
if xmlNewInputFromFile() fails.
* xmlreader.c: avoid using _private for the node or document
elements.
Daniel
* include/libxml/nanohttp.h include/libxml/parserInternals.h
include/libxml/xmlIO.h nanohttp.c parserInternals.c xmlIO.c:
Fixed the HTTP<->parser interraction, which should fix 2 long
standing bugs #104790 and #124054 , this also fix the fact that
HTTP error code (> 400) should not generate data, we usually
don't want to parse the HTML error information instead of the
resource looked at.
Daniel
* genChRange.py, chvalid.def, chvalid.c, include/libxml/chvalid.h:
new files for a different method for doing range validation
of character data.
* Makefile.am, parserInternals.c, include/libxml/Makefile.am,
include/libxml/parserInternals.h: modified for new range method.
* catalog.c: small enhance for warning message (using one
of the new range routines)
* HTMLparser.c c14n.c catalog.c error.c globals.c parser.c
parserInternals.c relaxng.c valid.c xinclude.c xmlIO.c xmlregexp.c
xmlschemas.c xpath.c xpointer.c include/libxml/globals.h
include/libxml/parser.h include/libxml/valid.h
include/libxml/xmlerror.h: Setting up the framework for structured
error reporting, touches a lot of modules, but little code now
the error handling trail has been cleaned up.
Daniel
* parserInternals.c parser.c valid.c include/libxml/parserInternals.h:
more cleanup of error handling in parserInternals, sharing the
routine for memory errors.
Daniel
* HTMLparser.c Makefile.am legacy.c parser.c parserInternals.c
include/libxml/xmlerror.h: more code cleanup, especially around
error messages, the HTML parser has now been upgraded to the new
handling.
* result/HTML/*: a few changes in the resulting error messages
Daniel
* Makefile.am: changed 'make tests' to use a concise output,
scrolling to see where thing broke wasn't pleasant
* configure.in: some beta4 preparation, but not ready yet
* error.c globals.c include/libxml/globals.h include/libxml/xmlerror.h:
new error handling code, last error informations are stored
in the parsing context or a global variable, new APIs to
handle the xmlErrorPtr type.
* parser.c parserInternals.c valid.c : started migrating to the
new error handling code, it's a royal pain.
* include/libxml/parser.h include/libxml/parserInternals.h:
moved the definition of xmlNewParserCtxt()
* parser.c: small potential buffer access problem in push code
provided by Justin Fletcher
* result/*.sax result/VC/PENesting* result/namespaces/*
result/valid/*.err: some error messages were sligthly changed.
Daniel
* HTMLparser.c Makefile.am configure.in legacy.c parser.c
parserInternals.c testHTML.c xmllint.c include/libxml/HTMLparser.h
include/libxml/parser.h include/libxml/parserInternals.h
include/libxml/xmlversion.h.in: added a new configure
option --with-push, some cleanups, chased code size anomalies.
Now a library configured --with-minimum is around 150KB,
sounds good enough.
Daniel
* Makefile.am: cleanup, creating a new legacy.c module,
made sure make tests ran in reduced conditions
* SAX.c SAX2.c configure.in entities.c globals.c parser.c
parserInternals.c tree.c valid.c xlink.c xmlIO.c xmlcatalog.c
xmlmemory.c xpath.c xmlmemory.c include/libxml/xmlversion.h.in:
increased the modularization, allow to configure out
validation code and legacy code, added a configuration
option --with-minimum compiling only the mandatory code
which then shrink to 200KB.
Daniel
* parserInternals.c: fixed a backward compatibility problem
when formatting "deprecated SAXv1 function ignorableWhitespace"
could be reproduced by xmllint --format
Daniel
* configure.in: preparing a beta3 solving the ABI problems
* globals.c parser.c parserInternals.c testHTML.c HTMLparser.c SAX.c
include/libxml/globals.h include/libxml/SAX.h: make sure the
global variables for the default SAX handler are V1 ones to
avoid ABI compat problems.
* xmlreader.c: cleanup of uneeded code
* hash.c: fix a comment
Daniel
* parserInternals.c: small fix from Rob Richards for input filename
* xmllint.c: fixes for --repeat and --memory/--stream for speed tests
* xmlIO: adding a guard in one function
Daniel
* SAX2.c parser.c parserInternals.c xmlreader.c: started messing
seriously with per-document dict and element and attribute nodes
reuse in the xmlReader. This seems to lead to an interesting
speedup of the xmlReader already.
Daniel
* parser.c include/libxml/xmlerror.h: factoring of more
error handling code, serious size reduction and more lisibility
of the resulting code.
* parserInternals.c parser.c include/libxml/parserInternals.h
include/libxml/parser.h: changing the way VC:Proper Group/PE Nesting
checks are done, use a counter for entities. Entities where freed and
reallocated at the same address failing the check.
* tree.c: avoid a warning
* result/valid/* result/VC/*: this slightly changes some validation
error messages.
Daniel
