IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
This fixes https://sft.its.cern.ch/jira/browse/ROOT-9672 by having
cling::Interpreter::DeclareCFunction return the transaction containing the
compiled code.
With the previous code, cling::Interpreter::compileFunction will get confused by
transaction created during the callbacks executed during the
cling::IncrementalParser::commitTransaction of the main transaction.
Reproducer:
With a main composed of 'only':
int main(int argc, char ** argv)
{
char const * class_string = (argc == 2) ? argv[1] : "std::vector<int>";
auto const result [[gnu::unused]] = TClass::GetClass(class_string);
return 0;
}
which is a representation of real use case (in a more complex setup) in ART.
We were getting:
Error in <TClingCallFunc::make_wrapper>: Failed to compile
==== SOURCE BEGIN ====
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wformat-security"
__attribute__((used)) extern "C" void __cf_0(void* obj, int nargs, void** args, void* ret)
{
if (ret) {
(*(TStreamerInfo**)ret) = new TStreamerInfo();
return;
}
else {
new TStreamerInfo();
return;
}
}
#pragma clang diagnostic pop
==== SOURCE END ====
Error in <TClingCallFunc::ExecT>: Called with no wrapper, not implemented!
Error in <TVirtualStreamerInfo::Factory>: The plugin handler for TVirtualStreamerInfo was found but failed to create the factory object!
The reason is that during TClingCallFunc::make_wrapper, the call to cling::Interpreter::compileFunction ends with:
if (const llvm::GlobalValue* GV
= getLastTransaction()->getModule()->getNamedValue(name))
However in the 'broken' case, the getLastTransaction does not return the transaction for the code being compiled by DeclareCFunction but instead the one used/created at:
#0 cling::IncrementalParser::endTransaction (this=0x4a2980, T=0x8c0fb0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:345
#1 0x00007fffeebc7899 in cling::Interpreter::PushTransactionRAII::pop (this=0x7fffffffcb00) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:111
#2 0x00007fffeebc785e in cling::Interpreter::PushTransactionRAII::~PushTransactionRAII (this=0x7fffffffcb00, __in_chrg=<optimized out>)
at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:106
#3 0x00007fffeebeb659 in cling::LookupHelper::findScope (this=0x4a9dd0, className=..., diagOnOff=cling::LookupHelper::NoDiagnostics, resultType=0x7fffffffcd08, instantiateTemplate=false)
at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/LookupHelper.cpp:466
#4 0x00007fffeeabe0df in TCling::CheckClassInfo (this=0x4a0550, name=<optimized out>, autoload=<optimized out>, isClassOrNamespaceOnly=<optimized out>)
at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TCling.cxx:3630
#5 0x00007ffff7c3040d in TClass::Init (this=this@entry=0xdafd20, name=name@entry=0x7ffff7cb7638 "TGlobal", cversion=cversion@entry=2, typeinfo=typeinfo@entry=0x7ffff7d8b6d8 <typeinfo for TGlobal>, isa=isa@entry=0x477430,
dfil=dfil@entry=0x7ffff7cb8cab "TGlobal.h", ifil=<optimized out>, dl=<optimized out>, il=<optimized out>, givenInfo=<optimized out>, silent=<optimized out>)
at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TClass.cxx:1431
#6 0x00007ffff7c3a1b8 in TClass::TClass (this=0xdafd20, name=0x7ffff7cb7638 "TGlobal", cversion=<optimized out>, info=..., isa=0x477430, dfil=0x7ffff7cb8cab "TGlobal.h",
ifil=0x7ffff7cccf88 "/local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TGlobal.cxx", dl=27, il=25, silent=false) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TClass.cxx:1273
#7 0x00007ffff7c3a72a in ROOT::CreateClass (cname=0x7ffff7cb7638 "TGlobal", id=id@entry=2, info=..., isa=isa@entry=0x477430, dfil=dfil@entry=0x7ffff7cb8cab "TGlobal.h",
ifil=ifil@entry=0x7ffff7cccf88 "/local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TGlobal.cxx", dl=27, il=25) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TClass.cxx:5607
#8 0x00007ffff7c4b552 in ROOT::Internal::TDefaultInitBehavior::CreateClass (il=25, dl=27, ifil=0x7ffff7cccf88 "/local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TGlobal.cxx", dfil=0x7ffff7cb8cab "TGlobal.h",
isa=0x477430, info=..., id=2, cname=<optimized out>, this=0x7ffff7da7508 <ROOT::Internal::DefineBehavior(void*, void*)::theDefault>) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/Rtypes.h:176
#9 ROOT::TGenericClassInfo::GetClass (this=0x7ffff7dab660 <ROOT::GenerateInitInstanceLocal(TGlobal const*)::instance>) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TGenericClassInfo.cxx:250
#10 0x00007ffff7b1a2d8 in TGlobal::Class () at /home/pcanal/root_builds/v6-14-00-patches/opt/core/base/G__Core.cxx:17156
#11 0x00007ffff7ac01de in TGlobal::IsA (this=0xee3bc0) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/TGlobal.h:48
#12 TGlobal::CheckTObjectHashConsistency (this=0xee3bc0) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/TGlobal.h:48
#13 0x00007ffff7be9dcd in TObject::CheckedHash (this=0xee3bc0) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/TObject.h:314
#14 THashTable::GetCheckedHashValue (this=0xe65a20, obj=0xee3bc0) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/THashTable.h:94
#15 THashTable::Add (this=0xe65a20, obj=0xee3bc0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/cont/src/THashTable.cxx:96
#16 0x00007ffff7be6bf1 in THashList::AddLast (this=this@entry=0x5be690, obj=obj@entry=0xee3bc0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/cont/src/THashList.cxx:100
#17 0x00007ffff7c4e0d1 in TListOfDataMembers::AddLast (this=0x5be690, obj=0xee3bc0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TListOfDataMembers.cxx:103
#18 0x00007ffff7ab8785 in TList::Add (obj=0xee3bc0, this=0x5be690) at /home/pcanal/root_builds/v6-14-00-patches/opt/include/TList.h:87
#19 TROOT::GetListOfGlobals (this=0x7ffff7da7a60 <ROOT::Internal::GetROOT1()::alloc>, load=load@entry=false) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/base/src/TROOT.cxx:1767
#20 0x00007fffeeab1058 in TCling::HandleNewDecl (this=0x4a0550, DV=0xedf238, isDeserialized=isDeserialized@entry=true, modifiedTClasses=...) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TCling.cxx:555
#21 0x00007fffeeabb785 in TCling::UpdateListsOnCommitted (this=0x4a0550, T=...) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TCling.cxx:6115
#22 0x00007fffeebd0103 in cling::MultiplexInterpreterCallbacks::TransactionCommitted (this=0x57fe20, T=...) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/MultiplexInterpreterCallbacks.h:76
#23 0x00007fffeed05d71 in cling::IncrementalParser::commitTransaction (this=0x4a2980, PRT=..., ClearDiagClient=true) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:532
#24 0x00007fffeed06399 in cling::IncrementalParser::Compile (this=0x4a2980, input=..., Opts=...) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/IncrementalParser.cpp:663
#25 0x00007fffeebcbc4e in cling::Interpreter::DeclareInternal (this=0x4a0f30, input=..., CO=..., T=0x7fffffffd680) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:1195
#26 0x00007fffeebca8e8 in cling::Interpreter::declare (this=0x4a0f30, input=..., T=0x7fffffffd680) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:823
#27 0x00007fffeebcb560 in cling::Interpreter::DeclareCFunction (this=0x4a0f30, name=..., code=..., withAccessControl=true) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:1096
#28 0x00007fffeebcb862 in cling::Interpreter::compileFunction (this=0x4a0f30, name=..., code=..., ifUnique=false, withAccessControl=true)
at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/interpreter/cling/lib/Interpreter/Interpreter.cpp:1140
#29 0x00007fffeeafb83c in TClingCallFunc::compile_wrapper (withAccessControl=true, wrapper=..., wrapper_name=..., this=0xcf3c10) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TClingCallFunc.cxx:270
#30 TClingCallFunc::make_wrapper (this=this@entry=0xcf3c10) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TClingCallFunc.cxx:1096
#31 0x00007fffeeafbcb8 in TClingCallFunc::IFacePtr (this=this@entry=0xcf3c10) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TClingCallFunc.cxx:2233
#32 0x00007fffeeafbe83 in TClingCallFunc::ExecT<long> (address=0x0, this=0xcf3c10) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TClingCallFunc.cxx:2045
#33 TClingCallFunc::ExecInt (this=0xcf3c10, address=0x0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/metacling/src/TClingCallFunc.cxx:2065
#34 0x00007ffff7c56e8d in TMethodCall::Execute (this=0xd97710, object=<optimized out>, retLong=@0x7fffffffd958: 0) at /local2/pcanal/cint_working/rootcling/v6-14-00-patches/core/meta/src/TMethodCall.cxx:457
#35 0x0000000000401009 in TMethodCall::Execute(long&) ()
#36 0x00000000004010ea in long TPluginHandler::ExecPluginImpl<>() ()
#37 0x000000000040106d in long TPluginHandler::ExecPlugin<>(int) ()
#38 0x0000000000400e21 in mytest() ()
#39 0x0000000000400e92 in main ()
This is the final version of "printValue" discussion.
We agreed that printValue interface should be altered to ToString
interface, which can be invoked `gInterpreter->ToString(XYZ)`. (ToString is in TCling and toString is in Interpreter :D)
This patch contains:
- Implementation of toString in Interpreter.cpp
- Re-Implementation of ClingPrintValue to use ToString because I changed to use Evaluate some time ago.
- Removing of RVec version of printValue which wasn't used at all
- Fix test/vecops_rvec.cxx, printValue is never supposed to be called by a normal user.
Calling VirtualQuery() is very expansive. For example, the testUnfold5a.C tutorial (interpreted) times out after 3600 seconds, and runs in about 60 seconds without calling VirtualQuery(). So just bypass it and return true for the time being
Visual Studio 2017 doesn't use the registry anymore, so no need to try to find keys. This prevent picking wrong directories in the case there is another version installed on the computer, and prevent also the following errors when starting ROOT:
RegQueryValueEx: returned 2: The system cannot find the file specified.
RegOpenKeyEx: returned 2: The system cannot find the file specified.
The operations done by the LookupHelper are costly in both memory and
performance. Almost every operation requires memory allocation and parsing
of often non-trivial C++ code.
Unfortunately, the LookupHelper is used very intensively by rootcling and
ROOT. The callers usually do not use any caching mechanisms and redo the
expensive operations over and over even though the answer is known to be
the same as before. For instance, building the dictionary of shows:
```
MathCore:
Cached entries: 217
Total parse requests: 54051
Cache hits: 53834
TreePlayer:
Cached entries: 183
Total parse requests: 57697
Cache hits: 57514
```
This patch introduces the first set of caching functionality. In
particular, each LookupHelper::find* function allocates a memory buffer
which is then stored in the clang::SourceManager. We hash the buffer
content and keep a mapping between a hash and FileID and next time we
encounter the same content we do not allocate a new FileID but reuse the
old one. We see decrease in memory footprint by 7% for non-cxxmodules ROOT.
For cxxmodules we see significant reduction of the pcm sizes (by half)
which translates into rss improvements:
```
master before:
cpu time = 0.291462 seconds
sys time = 0.064409 seconds
res memory = 345.816 Mbytes
vir memory = 573.508 Mbytes
master after:
cpu time = 0.235828 seconds
sys time = 0.098327 seconds
res memory = 260.012 Mbytes
vir memory = 377.945 Mbytes
```
Patch by Yuka Takahashi and me.
Clang allows third party shared libraries to provide user-defined
extensions. For example, a custom libTemplateInstantiation.so can
visualize all template instantiation chains in clang. To enable it
one needs to pass a set of options such as -fplugin.
Cling should be able to inherently work with clang plugins. However,
cling still does not make full use of the clang driver where the plugin
setup is handled.
This patch enables plugins in cling and extends them in some aspects.
In particular, cling allows loading of plugins from shared libraries
but also if they are linked to the same library where cling is. This is
very useful in cases where cling runs itself in a shared library (eg
libCling). Users of libCling (such as ROOT) prefer to keep all llvm and
clang related symbols local to avoid symbol clashes if there is another
version of clang and llvm linked against a package. This can be done by
dlopen-ing libCling with RTLD_LOCAL visibility mode. Then the only way
for clang plugins to work in this scenario is to be linked to libCling.
Patch by Aleksandr Efremov and me.
Problem:
If a c++ statement with a concrete template parameter will type in, cling pass the (unwrapped) input line and the specialization of the statement, generated by compilerinstance to the clang nvptx. This causes an explicit-specialization-after-instantiation-error.
Solution:
Template specialization declaration will be save two times at a transaction. Once with the type kCCIHandleCXXImplicitFunctionInstantiation and once with the type kCCIHandleTopLevelDecl. To avoid sending a template specialization to the clang nvptx and causing an explicit-specialization-after-instantiation-error it have to check, which kCCIHandleTopLevelDecl declaration is also a kCCIHandleCXXImplicitFunctionInstantiation declaration.
As input of clang nvptx there are two possible sources. The raw input of `IncrementalParser::parseInternal()` or the AST-Printer of the `llvm::module`. Now, to decide, which input source should be use, I use the value `Transaction::ConsumerCallInfo::kCCIHandleTopLevelDecl`. This allows more C++ constructs to use.
Fixed a small bug. The AST-printer prints `setValueNoAlloc()` without the scope `cling::runtime::internal`. This caused an error, because clang could not find the declaration in the global space. The solution are dummy declaration in the global space.
At the moment, templates doesn't works.
- little changes at comments and code style
- try to use const in IncrementalCUDADeviceCompiler, where is possible
- move CUDA device code compiler instance to IncrementalParser
- change the members of CuArgs to const and adjust the setCuArgs method
- use std::vector<string> instead llvm::Smallvector<const char *> to build argv for executeAndWait
- improve the error messages of generatePCH(), generatePTX() and generateFatbinary()
- replace m_Counter with a copy in IncrementalCUDADeviceCompiler to avoid involuntary changes
Before this commit, the CUDA device code Compiler class overwrited source code .cu files, if the translation get wrong. Now, it renames the faulty files, so you can check, which source code cause compiler errors.
Before, it was not possible to find the clang++, which is contained in the cling, if we don't start the clang from the bin folder ('./cling -xcuda'). Now, for example it is possible to start the cling with 'bin/cling -xcuda' .
Fix a Bug, which avoid to start './cling -xcuda -fsyntax-only'.
In some cases, the path of the cling temp folder contains some non printable chars at the end.
Change the handling of the path string, to solve this problem.
Now, it is possible to declare variables on the prompt, which are visible for other statements.
The problems was, that cling wrapped all statements in a function, to get valid input. So, every variable is just visible inside in its own wrapper function. To solve the problem, cling change the local variable declaration to a global declaration.
The implementation for the CUDA compiler checks, if the unwrapping is happened. If it happened, the c++ code of the unwrapped variable declaration (AST-printer) instead the raw input will be written to the .cu-file.
At the moment, to extend the AST-tree of the device code, we use PCH-files to extend the exist device code with new lines of code. In detail, if we want to create a new PTX-file, we use the CUDA code (.cu file) and a PCH-file with the existing AST as input and generate an new PCH-file, which contains the whole AST. Then, the PCH-file will compiled to a PTX-file.
A bug in clang prevent, that we can’t generate more than 5 new PCH-files. The bug is not easy to fix, so I write a small workaround. Instead using a PCH-file, which contains the AST, we generate a new complete AST from all .cu-files every time.
The workaround is temporary and should removed, if clang is patched.
Now, it is possible to set some arguments of the clang nvptx and fatbinary via arguments at cling start. The arguments are filtered. So not every argument is possible at the moment. The Arguments can’t changed during runtime, because the PCH-files forbid it. For Example, the calng nvptx use the optimization level, which is set at start of cling.
At the moment, the debug options of clang nvptx are simple. If any debug option is detected, just a -g will add to the clang nvptx.
Additional PTX options for clang nvptx doesn’t works at the moment. There is a problem at parsing at the start of cling.
I replaced copies of the include paths with a pointer to the headerSearchOptions. Now, explicit handling of the include paths is not more necessary. Add include paths, which was declared via argument at start also works.
The class IncrementalCUDADeviceCompiler use external tools to generate PTX and cuda fatbin files. It runs the tools clang and fatbinary via llvm::sys::ExecuteAndWait. The class also handle to include new code in existing code. The steps of the compiler pipeline are:
- clang: CUDA C++ + previous PCH -> PCH
- clang: PCH -> PTX
- fatbinary: PTX -> fatbin
There is no selection of code. Every input of the cling will pass to the IncrementalCUDADeviceCompiler.
Now, it is possible to define functions with c++ attributes without the .rawInput mode. For example functions like `[[ noreturn ]] foo() { ... }` or `[[deprecated]] [[nodiscard]] int bar(){ … }`.
This reverts commit 011aa8200277cd31957e222afd9b37415458b31f.
This is a revert of revert. I reverted the first commit because adding
"." to prebuiltmodulepath was causing failure in runtime modules, but
now we're skipping "." in TCling::LazyFunctionCreatorAutoloadForModule so
doesn't matter even if we have ".".
We had test failures in runtime nightlies such as this one:
https://epsft-jenkins.cern.ch/view/ROOT/job/root-nightly-runtime-cxxmodules/95/BUILDTYPE=Debug,COMPILER=gcc62,LABEL=slc6/testReport/junit/projectroot.roottest.root.math/smatrix/roottest_root_math_smatrix_testKalman/
Failures were due to what @pcanal commented in #2135, that some so files in
roottest doesn't have external linkage. (It means that if you call
dlopen(libfoo.so), linux kernel can't find dependency libraries and it
emits "undefined symbol" error when they try to initialize global
variables in libfoo.so but couldn't find symbol definition)
With pch, rootmap files were providing information about the depending library.
However we stopped generating rootmap files in #2127 and that's why we
got these failures. To fix this issue, I implemented a callback to
TCling which gets called when DynamicLibraryManager fails. The callback
pass error message to TCling and it handles message if it contains "undefined error".
This reverts commit 5298b418eec4129351888f41cb7c3bfc90161e22.
This commit was mistakenly committed. PR was opened in #1730, but it was
closed and moved to #1761. I didn't notice this and created another PR
in #1980.
This change was causing 100+ failures in runtime cxxmodules nightlies.
(Eg. https://epsft-jenkins.cern.ch/job/root-pullrequests-build/29183/testReport/junit/projectroot/runtutorials/tutorial_fit_FittingDemo/)
We want to have **proper** PrebuildModulesPaths which information were
extracted from LD_LIBRARY_PATH and DYLD_LIBRARY_PATH, not a random ".".
Because of this commit, we were trying to autoload libraries generated
by roottest on-demand (for example "./h1analysisTreeReader_C.so") This
is not an intentional behavior, these autogenerated libraries are
already loaded by roottest and what we want to do is to load **proper**
libraries like libHist.so instead.
In previous allmodules&autoloading patch, we used callback from
DeserializationListener to get Decl and loaded corresponding libraries.
It worked, but the performance was bad because ROOT was loading
excessive libraries.
In this patch, we use TCling::LazyFunctionCreatorAutoloadForModule. This
function gets callback when "mangled_name" was not found in loaded
libraries thus we have to the load corresponding library and lookup
again.
I used unordered_map to store mangled identifier and library pair. I'm
doing an optimization by hashing mangled name and storing library not by
name but by uint8 and hold uint8-name information in another vector.
Also tried std::map but unorderd_map was more performant. There are
better hash table like:
https://probablydance.com/2018/05/28/a-new-fast-hash-table-in-response-to-googles-new-fast-hash-table/
we can try to use them if this part gets crucial.
With this patch:
```
Processing tutorials/hsimple.C...
hsimple : Real Time = 0.04 seconds Cpu Time = 0.03 seconds
(TFile *) 0x562b37a14fe0
Processing /home/yuka/CERN/ROOT/memory.C...
cpu time = 0.362307 seconds
sys time = 0.039741 seconds
res memory = 278.215 Mbytes
vir memory = 448.973 Mbytes
```
W/o this patch:
```
Processing tutorials/hsimple.C...
hsimple : Real Time = 0.08 seconds Cpu Time = 0.07 seconds
(TFile *) 0x5563018a1d30
Processing /home/yuka/CERN/ROOT/memory.C...
cpu time = 1.524314 seconds
sys time = 0.157075 seconds
res memory = 546.867 Mbytes
vir memory = 895.184 Mbytes
```
So it improves time by 4x times and memory by 2x.
