8c31581728
Improved logging
...
It should better indicate when and why it stops running.
Also added log about finishing processing saved data when no writer thread is used.
2020-01-30 13:32:42 +03:00
6fecac06f8
Don't fill empty record fields preemptively
...
This change should reduce memory footprint of application
2020-01-29 16:36:56 +03:00
17d6a6489c
Don't store record names in the record itself
2020-01-29 14:47:27 +03:00
237f504efe
Only send signal for data writing when reaching specified amount of data
2020-01-28 11:20:45 +03:00
3f295f3e05
auparse: flush feed and release memory
...
This might produce some bad data if lines of data in input feed is incorrectly split
2020-01-28 11:20:45 +03:00
d8bde76fd2
Make audit data buffer size unconfigurable
2020-01-27 13:07:59 +03:00
3479a02ac2
Reduce objects copying
2020-01-27 13:07:59 +03:00
5dfb70a328
Don't stop writer thread before all data is processed
2020-01-27 13:07:59 +03:00
972da5776a
Fix memory leak from auparse_get_node
2020-01-27 13:07:59 +03:00
62adea5afd
Send data to database only in chunks exceeding limit count if one is specified
2020-01-27 13:07:59 +03:00
f0fe0f5a61
Don't store logs internally if logging to file is disabled
2020-01-27 13:07:59 +03:00
1520830a97
Don't pollute log while emptying data pipe and no data is present
2020-01-27 13:07:59 +03:00
7a04b13645
Add information about date and time into log
2020-01-27 13:07:59 +03:00
ddcdfa9c1c
Improve logging messages for POLLHUP and POLLERR conditions
...
Don't report EINTR from poll
2020-01-27 13:07:59 +03:00
6677b661c2
Save audit record filename into audit record class
...
This will allow to remove files even if file naming scheme is changed
2020-01-27 13:07:59 +03:00
e54ea41414
Read previously saved data when application is launched
2020-01-27 13:07:59 +03:00
aaf2d3b886
Implement audit data removal
2020-01-27 13:07:59 +03:00
a42ec11de1
Serialize and save data before sending it to write thread
2020-01-27 13:07:59 +03:00
be7e252971
Record all unknown fields into special "unknown_field" array
2020-01-27 13:07:59 +03:00
149a7e7a44
Move some functions to utils file
2020-01-27 13:07:59 +03:00
8e2910ab94
Add bufferization of data and separate writer thread
...
Add time and count limit and configuration options for them.
If no limits are set, separate thread is not used.
2020-01-27 13:07:59 +03:00
36891f1273
Move audit data convertion into separate functions
2020-01-27 13:07:59 +03:00
1895289a87
Add buffered logging and new option for logfile location
2020-01-27 13:07:59 +03:00
0ea79509b8
Add conversion from field type to string and use it to improve logging
2020-01-27 13:07:59 +03:00
517d5d87c4
Properly process NULL strings
2020-01-27 13:07:59 +03:00
0e852e9ad3
Don't use Nested clickhouse type, flatten tables.
2020-01-27 13:07:59 +03:00
853818b7ab
Also exit if some error with stdin is encountered
2020-01-27 13:07:59 +03:00
9906c2e5a4
Sanitize column names
2020-01-27 13:07:59 +03:00
c6b1330fb5
Write all data into database
2020-01-27 13:07:59 +03:00
ef3b71bf27
Catch and process exceptions in audit callback
2020-01-27 13:07:59 +03:00
575ca79ff1
Populate empty fields
2020-01-27 13:07:59 +03:00
7ed7d49a25
Add function for columns generation from record field class
2020-01-27 13:07:59 +03:00
508ca46a67
Update string_array type to use nested arrays instead of single array
2020-01-27 13:07:59 +03:00
98bc48492f
audit record: create and reuse fields
2020-01-27 13:07:59 +03:00
4de04f70b9
Improve audit callback: start creating audit record to be added to DB later
2020-01-27 13:07:59 +03:00
3c09000edc
Implement string_array type
...
Update AbstractRecordField interface to match new use-case
2019-12-17 14:34:19 +03:00
3ba3561664
Implement warnings and mark all optional fields as Nullable
2019-12-17 14:34:19 +03:00
d7ac138531
Implement clickhouse database creation
2019-12-17 14:33:32 +03:00
cfd18dd0d9
Initial processing of audit types added
2019-12-17 14:32:09 +03:00
7f1ed9da8f
Add clickhouse config parsing and initial audit setup
2019-12-16 16:42:52 +03:00
35d96682eb
Initial commit
2019-12-16 16:42:40 +03:00
