IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
For https://bugzilla.gnome.org/show_bug.cgi?id=666491
Reported by Matt Budd <matt.budd@gmail.com>, the added support
for VS 2010 broke older version 2005 and 2008 because it assumed
some of the defines where present in all versions, fix that
to check the version of VS
tsan reported that rand() is not thread safe, so create
a thread safe wrapper, use rand_r() if available.
Consolidate the function, initialization and cleanup in
dict.c and make sure it is initialized in xmlInitParser()
For https://bugzilla.gnome.org/show_bug.cgi?id=643148
Reported by Bill Clarke <llib@computer.org>, it used a global variable
as a counter for the input id and this was not thread safe. To avoid the
race without adding unneeded locking in the parser path, move the id to
the parser context instead.
On Fri, May 11, 2012 at 9:10 AM, Daniel Veillard <veillard@redhat.com> wrote:
> Hi Conrad,
>
> that's interesting ! I was initially afraid of a sudden explosion of
> memory allocations for building a tree since by default buffers tend to
> "waste" memory by using doubling allocations, but that's not the case.
> xmllint --noout doc/libxml2-api.xml
> when compiled with memory debug produce
>
> paphio:~/XML -> cat .memdump
> MEMORY ALLOCATED : 0, MAX was 12756699
>
> and without your patch 12755657, i.e. the increase is minimal.
Heh, I thought that too. Actually you're looking at the result with XML_ALLOC_EXACT! This
is because EXACT adds 10bytes "spare" on each alloc, and that interestingly wastes about the
same amount of space as XML_ALLOC_DOUBLEIT on this example (see below).
So it turns out that the default realloc() on my system actually handles this case really
well — and I guess that all the time in xmlRealloc() was actually in xmlStrlen, not the
underlying realloc() after all (sorry for misleading you). If you replace the realloc()
with a bad one (like valgrind's), then the performance degrades severely.
This patch implements a HYBRID allocator which has the behaviour you describe (it's
like EXACT to start with, though without the spare 10 bytes; and switches to DOUBLEIT
after 4kb) — that gets the memory back down to 12755657, with no noticeable impact on the
performance of the synthetic pathological example under valgrind.
In summary:
max_memory on ./xmllint --noout doc/libxml2-api.xml,
valgrind time on https://gist.github.com/2656940
max_memory valgrind time
before | 12755657 | 29:18.2
EXACT | 12756699 | 2:58.6 <-- this is the state after the first patch.
DOUBLEIT | 12756727 | 0:02.7
HYBRID | 12755754 | 0:02.7 <-- this is the state with both patches.
>
> There is also the cost of creating the buffers all the time.
> I need to read the code and check but I may be interested in an hybrid
> approach where we switch to buffer only when the text node starts to
> become too big (4k would remove nearly all usuall types of "document"
> usage, i.e. not blocks of data)
I tried to avoid too much buffer creation by introducing the xmlBufferDetach function,
which allows re-using one buffer to construct many strings. It's maybe a bit of a "hack"
in API terms though I thought the gains would be worth it.
Conrad
------8<------
To keep memory usage tight in normal conditions it's desirable to only
allocate as much space as is needed. Unfortunately this can lead to
problems when constructing a long string out of small chunks, because
every chunk you add will need to resize the buffer.
To fix this XML_ALLOC_HYBRID will switch (when the buffer is 4kb big)
from using exact allocations to doubling buffer size every time it is
full. This limits the number of buffer resizes to O(log n) (down from
O(n)), and thus greatly increases the performance of constructing very
large strings in this manner.
Hi Veillard and all,
Firstly, thanks for libxml: it's awesome!
I noticed recently that libxml was taking a surprisingly long time to perform some
operations (many minutes instead of milliseconds), and so I did some digging. It turns out
that the problem was caused by the realloc()ing done in xmlNodeAddContentLen() which can
be called many (many) times when assigning some content into a node.
For background, I'm dealing with XML that contains emails, these can have large
attachments (~6MB) which are base-64 encoded, line-wrapped at 78 chars, and each line ends
with . This means that xmlNodeAddContentLen() is being called about 200,000 times,
and so there are 200,000 reallocs of a 6MB string, which takes a while... (I put a synthetic
example of this at https://gist.github.com/2656940)
The attached patch works around that problem by using the existing buffer API to merge the
strings together before even creating the text node, this keeps the number of realloc()s
at a managable level.
I'd love feedback on the patch, and am happy to fix problems with it, or explore other
solutions if you think that this is barking up the wrong tree :).
Thanks,
Conrad
P.S. Should I create a bug for this too?
------8<------
Before this change xmlStringGetNodeList would perform a realloc() of the
entire new content for every XML entity in the assigned text in order to
merge together adjacent text nodes. This had the effect of making
xmlSetNodeContent O(n^2), which led to unexpectedly bad performance on
inputs that contained a large number of XML entities.
After this change the memory management is done by the buffer API,
avoiding the need to continually re-measure and realloc() the string.
For my test data (6MB of 80 character lines, each ending with )
this takes the time to xmlSetNodeContent from about 500 seconds to
around 50ms. I have not profiled smaller cases, though I tried to
minimize the performance impact of my change by avoiding unnecessary
string copying.
Signed-off-by: Conrad Irwin <conrad.irwin@gmail.com>
For https://bugzilla.gnome.org/show_bug.cgi?id=666491
This patch adds project files to compile and debug libxml2 using Visual
Studio 2010. Only few minor changes have been made to the actual source
code.
This patch also requires for the iconv package to be compiled with visual
studio 2010 which has been submitted to the iconv project (see:
https://savannah.gnu.org/bugs/?35088)
Since there is xmlTextReaderSchemaValidateCtxt() it seems like there
should be an equivalent RelaxNG function. The attached patch adds it.
The code is essentially the same as Schema implementation, but I'm
uncertain as to how to add things to the documentation and test suite:
there seems to be a lot of auto-generation going on.
For both XML and HTML, the document can provide an encoding
either in XMLDecl in XML, or as a meta element in HTML head.
This adds options to ignore those encodings if the encoding
is known in advace for example if the content had been converted
before being passed to the parser.
* parser.c include/libxml/parser.h: add XML_PARSE_IGNORE_ENC option
for XML parsing
* include/libxml/HTMLparser.h HTMLparser.c: adds the
HTML_PARSE_IGNORE_ENC for HTML parsing
* HTMLtree.c: fix the handling of saving when an unknown encoding is
defined in meta document header
* xmllint.c: add a --noenc option to activate the new parser options
In Windows 64 a socket is no more represented by an int,
this breaks the nanoftp API and nanoftp/nanohttp, the patch
changes this and fix the API for Win64
Regenerated the XML and documentation as a result too.
non destructive indentation option using spaces within markup
constructs and hence not modifying content
* include/libxml/xmlsave.h: new option
* xmlsave.c: some refactoring and new code for the new option
* xmllint.c: adds --pretty option where option 2 uses the new formatting
- include/libxml/HTMLparser.h: defines the new HTML parser option
HTML_PARSE_NODEFDTD
- HTMLparser.c: if option is set don't add a default DTD
- xmllint.c: add the corresponding --nodefdtd option in xmllint
* HTMLparser.c: new htmlParseElementInternal non recursive, with
htmlParseContentInternal and new function to handle node info
and element end.
* include/libxml/parser.h: add new stack for element info in parser
context
* parserInternals.c: fee element info stack
- include/libxml/xmlexports.h: restore export decoration otherwise
xsltproc and xmlsec crash
- libxml.h: define LIBXML_STATIC for static build
- configure.in: enable modules support for mingw* builds
- Makefile.am: flags for testdso if modules support enabled
xmlParseInNodeContext notices that the enclosing document is
an HTML document, so invoke the HTML parser for that fragment, and
the HTML parser finding a "<p>hello world!</p>" document automatically
augment it with defaulted <html> and <body>. This defaulting should
be turned off in the HTML parser for this to work, but there is no
such HTML parser option. There is an htmlOmittedDefaultValue global
variable that you could use, but really we should not rely on global
variable for processing options anymore, best is to add an
HTML_PARSE_NOIMPLIED.
* include/libxml/HTMLparser.h: add the HTML_PARSE_NOIMPLIED parser flag
* HTMLparser.c: do add implied element if HTML_PARSE_NOIMPLIED is set
* parser.c: add HTML_PARSE_NOIMPLIED to options for xmlParseInNodeContext
on HTML documents
* include/libxml/globals.h globals.c global.data: define a new global
variable (per thread) for structured error reporting, to not conflict
with generic one
* error.c: when defined use the structured error report over any generic
one
* HTMLparser.c: check when we see an head or a body tag and avoid
autogenerating them
* include/libxml/parser.h: the values for ctxt->html change depending
on the head or body tags being seen
* include/libxml/xmlexports.h: when compiling with mingw/MSYS or linking
to an precompiled library this _imp__xmlFree missing at runtime is a
common problem. Igor and various people faced it and this seems the
minimal fix for it, should resolve 590302 and 561340
* c14n.c include/libxml/c14n.h: adds support for C14N 1.1,
new flags at the API level
* runtest.c Makefile.am testC14N.c xmllint.c: add support in CLI
tools and test binaries
* result/c14n/1-1-without-comments/* test/c14n/1-1-without-comments/*:
add a new batch of tests
* include/libxml/parser.h include/libxml/xmlwriter.h
include/libxml/relaxng.h include/libxml/xmlversion.h.in
include/libxml/xmlwin32version.h.in include/libxml/valid.h
include/libxml/xmlschemas.h include/libxml/xmlerror.h: change
ATTRIBUTE_PRINTF into LIBXML_ATTR_FORMAT to avoid macro name
collisions with other packages and headers as reported by
Belgabor and Mike Hommey
daniel
svn path=/trunk/; revision=3827
* include/wsockcompat.h win32/Makefile.bcb xpath.c: fixes for
Borland/CodeGear/Embarcadero compilers by Eric Zurcher
Daniel
svn path=/trunk/; revision=3822
* configure.in doc/xml.html doc/*: preparing 0.7.3 release
* include/libxml/parserInternals.h SAX2.c: fix a typo in a name
Daniel
svn path=/trunk/; revision=3814
* include/libxml/parser.h include/libxml/xmlwriter.h
include/libxml/relaxng.h include/libxml/xmlversion.h.in
include/libxml/xmlwin32version.h.in include/libxml/valid.h
include/libxml/xmlschemas.h include/libxml/xmlerror.h:
port patch from Marcus Meissner to add gcc checking for
printf like functions parameters, should fix#65068
* doc/apibuild.py doc/*: modified the script accordingly
and regenerated
* xpath.c xmlmemory.c threads.c: fix a few warnings
Daniel
svn path=/trunk/; revision=3813
* include/libxml/xmlversion.h.in include/libxml/xmlmemory.h:
apply patch from Marcus Meissner to add gcc attribute alloc_size
should fix#552505
* doc/apibuild.py doc/* testapi.c: regenerate the API
* include/libxml/parserInternals.h: fix a comment problem raised
by apibuild.py
daniel
svn path=/trunk/; revision=3811
* include/libxml/parserInternals.h SAX2.c: add a new define
XML_MAX_TEXT_LENGHT limiting the maximum size of a single text
node, the defaultis 10MB and can be removed with the HUGE
parsing option
Daniel
svn path=/trunk/; revision=3808
* include/libxml/tree.h tree.c python/generator.py: adds
element traversal support
* valid.c: avoid a warning
* doc/*: regenerated
daniel
svn path=/trunk/; revision=3804
* include/libxml/xmlsave.h xmlsave.c: new options to serialize
as XML/HTML/XHTML and restore old entry point behaviours
Daniel
svn path=/trunk/; revision=3794
* include/libxml/tree.h tree.c: make a new kind of buffer where
shrinking and adding in head can avoid reallocation or full
buffer memmoves
* encoding.c xmlIO.c: use the new kind of buffers for output
buffers
Daniel
svn path=/trunk/; revision=3787
* parser.c include/libxml/parser.h: completely different fix for
the recursion detection based on entity density, big cleanups
in the entity parsing code too
* result/*.sax*: the parser should not ask for used defined versions
of the predefined entities
* testrecurse.c: automatic test for entity recursion checks
* Makefile.am: added testrecurse
* test/recurse/lol* test/recurse/good*: a first set of tests for
the recursion
Daniel
svn path=/trunk/; revision=3783
* include/libxml/xmlerror.h parser.c: a bit of cleanup and
added checks based on the regression tests of the xmlconf suite
Daniel
svn path=/trunk/; revision=3782
* include/libxml/parser.h parser.c xmllint.c: strengthen some
of the internal parser limits, add an XML_PARSE_HUGE option
to bypass them all. More internal parser limits will still need
to be added.
Daniel
svn path=/trunk/; revision=3777
* include/libxml/parser.h xinclude.c xmllint.c: patch based on
Wieant Nielander contribution to add the option of not doing
URI base fixup in XInclude
Daniel
svn path=/trunk/; revision=3775
* xmlreader.c: applied patch from Aswin to fix tree skipping
* include/libxml/entities.h entities.c: fixed a comment and
added a new xmlNewEntity() entry point
* runtest.c: be less verbose
* tree.c: space and tabs cleanups
daniel
svn path=/trunk/; revision=3774
* include/libxml/entities.h entities.c SAX2.c parser.c: rework
the patch to avoid some ABI issue with people allocating
entities structure directly
Daniel
svn path=/trunk/; revision=3773
