1
0
mirror of https://github.com/ansible/awx.git synced 2024-11-01 16:51:11 +03:00
Commit Graph

25304 Commits

Author SHA1 Message Date
Ryan Petrello
0f74a05fea
rsyslogd: ignore /dev/log when we load imuxsock 2020-04-14 11:34:58 -04:00
Ryan Petrello
dbe949a2c2
Merge pull request #6697 from chrismeyersfsu/fix-collection_tests
ensure last comma removed in select
2020-04-14 11:05:29 -04:00
chris meyers
ee18400a33 ensure last comma removed in select
* We strip out the json select fields in our tests since it is an sql
lite database underneath. Ideally, we would do something fancier, but we
aren't. In doing this stipping, we could strip the last element in the
projection list. This would result in an extra dangling comma. This
commit removes the danging comma in the projection list after the
removal of JSON projections.
2020-04-14 10:44:02 -04:00
softwarefactory-project-zuul[bot]
98a4e85db4
Merge pull request #6108 from rooftopcellist/rsyslog
Replace our external logging feature with Rsyslog

Reviewed-by: Ryan Petrello
             https://github.com/ryanpetrello
2020-04-14 13:40:41 +00:00
Ryan Petrello
f7f1bdf9c9
properly configure supervisorctl to point at the web volume mount 2020-04-13 21:56:52 -04:00
Ryan Petrello
69cf915a20
add rsyslogd block to the k8s supervisord config file 2020-04-13 20:25:53 -04:00
Ryan Petrello
9440785bdd
properly set the group on the rsyslog config 2020-04-13 19:46:34 -04:00
Christian Adams
ca7c840d8c Fix permissions on rsyslog.conf for k8s 2020-04-13 19:33:23 -04:00
softwarefactory-project-zuul[bot]
f85bcae89f
Merge pull request #6685 from marshmalien/fix-user-loading
Fix route bug in User view

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 20:00:25 +00:00
Christian Adams
a0e31b9c01 Map logging timeout value to healthchecktimeout for http in rsyslog config 2020-04-13 15:22:16 -04:00
softwarefactory-project-zuul[bot]
c414fd68a0
Merge pull request #6176 from Ladas/send_also_workflows_as_part_of_unified_jobs
Send also workflows as part of unified jobs and send all changes to jobs

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:41:36 +00:00
softwarefactory-project-zuul[bot]
2830cdfdeb
Merge pull request #6668 from nixocio/ui_refactor_users_functional
Modify Users component to be function based

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:35:57 +00:00
softwarefactory-project-zuul[bot]
07e9b46643
Merge pull request #6656 from jlmitch5/withoutWithRouter
excise withRouter from function components

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:35:53 +00:00
softwarefactory-project-zuul[bot]
1f01521213
Merge pull request #6651 from nixocio/ui_issue_5820
Rename SCM to Source Control

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:35:46 +00:00
Marliana Lara
8587461ac9
Fix loading bug in User view 2020-04-13 14:19:16 -04:00
nixocio
e54e5280f2 Modify Users component to be function based
Modify Users component to be function based.
2020-04-13 13:43:22 -04:00
softwarefactory-project-zuul[bot]
516a44ce73
Merge pull request #6662 from keithjgrant/5909-jt-launch-prompt-2
JT Launch Prompting (phase 2)

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 17:04:41 +00:00
Ryan Petrello
e52cebc28e rsyslogd: use %rawmsg-after-pri% instead of %msg%
after some prolonged RFC reading and tinkering w/ rsyslogd...

cpython's SysLogHandler doesn't emit RFC3164 formatted messages
in the format you'd expect; it's missing the ISO date, hostname, etc...
along with other header values; the handler implementation relies on you
to specify a syslog-like formatter (we've replaced all of this with our
own *custom* logstash-esque formatter that effectively outputs valid JSON
- without dates and other syslog header values prepended)

because of this unanticipated format, rsyslogd chokes when trying to
parse the message's parts;  AWX is emitting:

<priority>RAWJSON

...so the usage of `%msg%` isn't going to work for us, because rsyslog
tries to parse *all* of the possible headers (and yells, because it
can't find a date to parse):

see: https://www.rsyslog.com/files/temp/doc-indent/configuration/properties.html#message-properties

this is fine, because we don't *need* any of that message parsing
anyways; in the end, we're *just* interested in forwarding the raw
JSON/text content to the third party log handler
2020-04-13 11:44:00 -04:00
Ryan Petrello
bb5136cdae properly escape URL paths and querystrings for paths in logging settings 2020-04-13 11:44:00 -04:00
Ryan Petrello
b0db2b7bec add some exception handling for dealing with logging connection resets
when rsyslogd restarts due to config changes, there's a brief moment
where the socket will refuse connections on teardown; exception handling
is needed here to deal with that
2020-04-13 11:44:00 -04:00
Ryan Petrello
1000dc10fb an an rsyslogd config check to the logging test endpoint 2020-04-13 11:44:00 -04:00
Ryan Petrello
2a4b009f04 rsyslogd: use %rawmsg-after-pri% instead of %msg%
after some prolonged RFC reading and tinkering w/ rsyslogd...

cpython's SysLogHandler doesn't emit RFC3164 formatted messages
in the format you'd expect; it's missing the ISO date, hostname, etc...
along with other header values; the handler implementation relies on you
to specify a syslog-like formatter (we've replaced all of this with our
own *custom* logstash-esque formatter that effectively outputs valid JSON
- without dates and other syslog header values prepended)

because of this unanticipated format, rsyslogd chokes when trying to
parse the message's parts;  AWX is emitting:

<priority>RAWJSON

...so the usage of `%msg%` isn't going to work for us, because rsyslog
tries to parse *all* of the possible headers (and yells, because it
can't find a date to parse):

see: https://www.rsyslog.com/files/temp/doc-indent/configuration/properties.html#message-properties

this is fine, because we don't *need* any of that message parsing
anyways; in the end, we're *just* interested in forwarding the raw
JSON/text content to the third party log handler
2020-04-13 11:44:00 -04:00
Ryan Petrello
8cdd42307c clarify that logging username/password is only valid for HTTP/s 2020-04-13 11:44:00 -04:00
Ryan Petrello
269558876e only use a basic auth password for external logging if username is set 2020-04-13 11:44:00 -04:00
Ryan Petrello
bba680671b when writing the rsyslog config, do it post-commit
there's a race condition if we do this pre-commit where the correct
value isn't actually *persisted* to the database yet, and we end up
saving the *prior* setting values
2020-04-13 11:44:00 -04:00
Ryan Petrello
f70a76109c make rsyslog fall back to no-op if logging is disabled 2020-04-13 11:44:00 -04:00
Christian Adams
5d54877183 Add action to default rsyslog.conf so supervisor starts correctly the first time 2020-04-13 11:44:00 -04:00
Ryan Petrello
f7dac8e68d more external logging unit test fixups 2020-04-13 11:44:00 -04:00
Ryan Petrello
39648b4f0b fix up a few test and lint errors related to external logging 2020-04-13 11:44:00 -04:00
Christian Adams
b942fde59a Ensure log messages have valid json
- Fix messages getting contatenated at 8k
 - Fix rsyslog cutting off the opening brace of log messages
 - Make valid default conf and emit logs based on prescence of .sock and
 settings
2020-04-13 11:44:00 -04:00
Ryan Petrello
ce82b87d9f rsyslog hardening (fixing a few weird things we noticed) 2020-04-13 11:44:00 -04:00
Christian Adams
70391f96ae Revert rsyslog valid config to one that fails intentionally 2020-04-13 11:43:59 -04:00
Christian Adams
2329c1b797 Add rsyslog config to container from file for consistency 2020-04-13 11:43:59 -04:00
Christian Adams
470159b4d7 Enable innocuous but valid config for rsyslog if disabled 2020-04-13 11:43:59 -04:00
Christian Adams
e740340793 ConfigMap rsyslog conf files for k8 2020-04-13 11:43:59 -04:00
Christian Adams
4d5507d344 Add default rsyslog.conf without including /etc/rsyslog.conf 2020-04-13 11:43:59 -04:00
Christian Adams
d350551547 Tweaks to Test Button logic and cleans up flake8 and test failures 2020-04-13 11:43:59 -04:00
Christian Adams
7fd79b8e54 Remove unneeded logging sock variable 2020-04-13 11:43:59 -04:00
John Mitchell
eb12f45e8e add ngToast disable on timeout for log agg notifications, and disable test button until active test completes. 2020-04-13 11:43:59 -04:00
Christian Adams
fb047b1267 Add unit tests for reconfiguring rsyslog & for test endpoint 2020-04-13 11:43:59 -04:00
Christian Adams
d31c528257 Fix Logging settings "Test" button functionality 2020-04-13 11:43:59 -04:00
Christian Adams
996d7ce054 Move supervisor and rsyslog sock files to their own dirs under /var/run 2020-04-13 11:43:59 -04:00
Christian Adams
7040fcfd88 Fix container rsyslog dir permissions 2020-04-13 11:43:59 -04:00
John Mitchell
88ca4b63e6 update configure tower in tower test ui for log aggregator form 2020-04-13 11:43:59 -04:00
Shane McDonald
c0af3c537b Configure rsyslog to listen over a unix domain socket instead of a port
- Add a placeholder rsyslog.conf so it doesn't fail on start
 - Create access restricted directory for unix socket to be created in
 - Create RSyslogHandler to exit early when logging socket doesn't exist
 - Write updated logging settings when dispatcher comes up and restart rsyslog so they  take effect
 - Move rsyslogd to the web container and create rpc supervisor.sock
 - Add env var for supervisor.conf path
2020-04-13 11:43:59 -04:00
Christian Adams
f8afae308a Add rsyslog to supervisor for the task container
- Add proper paths for rsyslog's supervisor logs
 - Do not enable debug mode for rsyslogd
 - Include system rsyslog.conf, and specify tower logging conf when
   starting rsyslog.
2020-04-13 11:43:59 -04:00
Christian Adams
4cd0d60711 Properly handle logger paths and https/http configuration
- log aggregator url paths were not being passed to rsyslog
 - http log services like loggly will now truly use http and port 80
 - add rsyslog.pid to .gitignore
2020-04-13 11:43:59 -04:00
Christian Adams
955d57bce6 Upstream rsyslog packaging changes
- add rsyslog repo to Dockerfile for AWX installation
 - Update Library Notes for requests-futures removal
2020-04-13 11:43:59 -04:00
Ryan Petrello
589d27c88c POC: replace our external log aggregation feature with rsyslog
- this change adds rsyslog (https://github.com/rsyslog/rsyslog) as
  a new service that runs on every AWX node (managed by supervisord)
  in particular, this feature requires a recent version (v8.38+) of
  rsyslog that supports the omhttp module
  (https://github.com/rsyslog/rsyslog-doc/pull/750)
- the "external_logger" handler in AWX is now a SysLogHandler that ships
  logs to the local UDP port where rsyslog is configured to listen (by
  default, 51414)
- every time a LOG_AGGREGATOR_* setting is changed, every AWX node
  reconfigures and restarts its local instance of rsyslog so that its
  fowarding settings match what has been configured in AWX
- unlike the prior implementation, if the external logging aggregator
  (splunk/logstash) goes temporarily offline, rsyslog will retain the
  messages and ship them when the log aggregator is back online
- 4xx or 5xx level errors are recorded at /var/log/tower/external.err
2020-04-13 11:43:59 -04:00
softwarefactory-project-zuul[bot]
eafb751ecc
Merge pull request #6679 from ryanpetrello/fix-6675
skip non-files when consuming events synced from isolated hosts

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 15:36:42 +00:00