darktemplar/auditd-plugin-clickhouse
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
66 Commits 2 Branches 0 Tags 94 KiB
sisyphus
Commit Graph

66 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aleksei Nikiforov
ade304c040 20200207.3-alt1 
- Logged and ignored potential exceptions when saving data to temporary storage.
 2020-02-07 17:59:19 +03:00
Aleksei Nikiforov
390dd83733 Log and ignore exceptions on failure to write data to disk for temporary storage 2020-02-07 17:54:59 +03:00
Aleksei Nikiforov
7f28de5a99 20200130.3-alt1 
- Optimized memory consumption.
- Added new field to database.
- Minor logging improvements.
 2020-01-30 13:32:42 +03:00
Aleksei Nikiforov
7207939afe Add new field "port" to known database fields 2020-01-30 13:32:42 +03:00
Aleksei Nikiforov
8c31581728 Improved logging 
It should better indicate when and why it stops running.
Also added log about finishing processing saved data when no writer thread is used.
 2020-01-30 13:32:42 +03:00
Aleksei Nikiforov
6fecac06f8 Don't fill empty record fields preemptively 
This change should reduce memory footprint of application
 2020-01-29 16:36:56 +03:00
Aleksei Nikiforov
17d6a6489c Don't store record names in the record itself 2020-01-29 14:47:27 +03:00
Aleksei Nikiforov
69df9988ad 20200127-alt1 
- Fixed excessive logging of writes.
- Fixed memory consumption by logging entries if logging is disabled.
- Fixed memory leak in audit data parsing.
- Fixed issue when some data may be not attempted to be written to DB
  when separate writer thread is used.
- Improved performance by reducing data copying.
- Input data buffer size configuration is removed.
 2020-01-28 18:04:21 +03:00
Aleksei Nikiforov
893f013d27 Use std::make_shared for AbstractRecordField descendands creation 2020-01-28 18:04:21 +03:00
Aleksei Nikiforov
237f504efe Only send signal for data writing when reaching specified amount of data 2020-01-28 11:20:45 +03:00
Aleksei Nikiforov
3f295f3e05 auparse: flush feed and release memory 
This might produce some bad data if lines of data in input feed is incorrectly split
 2020-01-28 11:20:45 +03:00
Aleksei Nikiforov
d8bde76fd2 Make audit data buffer size unconfigurable 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
3479a02ac2 Reduce objects copying 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
5dfb70a328 Don't stop writer thread before all data is processed 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
972da5776a Fix memory leak from auparse_get_node 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
62adea5afd Send data to database only in chunks exceeding limit count if one is specified 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
f0fe0f5a61 Don't store logs internally if logging to file is disabled 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
1520830a97 Don't pollute log while emptying data pipe and no data is present 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
7ae97f57d7 20200116-alt1 
- New fields added to database table.
- All unknown fields encountered are saved into database.
- Fixed crash when processing NULL strings.
- Fixed warnings for some audit data types.
- Implemented logging.
- Implemented audit data serialization and temporary storing before sending
  to database.
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
1dc7f65269 Add new fields to known database fields 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
6fc2fe5320 Add serialization test 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
bb98d8809a Get rid of some dynamic_pointer_casts by updating return types 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
7a04b13645 Add information about date and time into log 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
ddcdfa9c1c Improve logging messages for POLLHUP and POLLERR conditions 
Don't report EINTR from poll
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
6677b661c2 Save audit record filename into audit record class 
This will allow to remove files even if file naming scheme is changed
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
e54ea41414 Read previously saved data when application is launched 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
aaf2d3b886 Implement audit data removal 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
e9cb80077d Implement audit record deserialization 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
a42ec11de1 Serialize and save data before sending it to write thread 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
be7e252971 Record all unknown fields into special "unknown_field" array 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
b1287924f2 Make logging compatible with logrotate tool and provide logrotate config 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
149a7e7a44 Move some functions to utils file 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
8e2910ab94 Add bufferization of data and separate writer thread 
Add time and count limit and configuration options for them.
If no limits are set, separate thread is not used.
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
36891f1273 Move audit data convertion into separate functions 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
1895289a87 Add buffered logging and new option for logfile location 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
0ea79509b8 Add conversion from field type to string and use it to improve logging 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
559944866f Fix AUPARSE_TYPE_ARCH and AUPARSE_TYPE_CAP_BITMAP checks 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
c99829625f Unclassified field type can be either string or integer 
Don't produce warnings for such types
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
517d5d87c4 Properly process NULL strings 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
ba27894d20 20191217-alt1 
- Initial build for ALT
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
c811c95797 Fix column names after tables flattening 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
a0ea1c1f26 Revert "Process nested structures as arrays" 
This reverts commit 0f41d6a19c18a2ead8b709fab515cdb9770422ae.
 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
0e852e9ad3 Don't use Nested clickhouse type, flatten tables. 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
853818b7ab Also exit if some error with stdin is encountered 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
f2c0a405dd Process nested structures as arrays 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
9906c2e5a4 Sanitize column names 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
10b9a9c159 Enable reconnect to clickhouse server by default 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
9d5a803603 Fix column types 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
c6b1330fb5 Write all data into database 2020-01-27 13:07:59 +03:00
Aleksei Nikiforov
ef3b71bf27 Catch and process exceptions in audit callback 2020-01-27 13:07:59 +03:00