1
0
mirror of https://github.com/ansible/awx.git synced 2024-10-30 13:55:31 +03:00
Commit Graph

24837 Commits

Author SHA1 Message Date
AlanCoding
558814ef3b tower_group relationships
rollback some module_utils changes
add runtime error for 404 type things
2020-03-19 13:53:08 -04:00
beeankha
ace5a0a2b3 Update module utils, part of collections conversion work 2020-03-19 13:53:08 -04:00
softwarefactory-project-zuul[bot]
8a917a5b70
Merge pull request #6343 from AlanCoding/fix_sanity
Fix sanity error

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-19 17:28:12 +00:00
softwarefactory-project-zuul[bot]
74ebb0ae59
Merge pull request #6290 from ryanpetrello/notification-host-summary-race
change when we send job notifications to avoid a race condition

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-19 15:36:24 +00:00
Ryan Petrello
d40a5dec8f
change when we send job notifications to avoid a race condition
success/failure notifications for *playbooks* include summary data about
the hosts in based on the contents of the playbook_on_stats event

the current implementation suffers from a number of race conditions that
sometimes can cause that data to be missing or incomplete; this change
makes it so that for *playbooks* we build (and send) the notification in
response to the playbook_on_stats event, not the EOF event
2020-03-19 10:01:52 -04:00
softwarefactory-project-zuul[bot]
a725778b17
Merge pull request #6327 from ryanpetrello/py2-minus-minus-cli
remove python2 support from awxkit

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-19 13:58:02 +00:00
softwarefactory-project-zuul[bot]
3b520a8ee8
Merge pull request #6341 from egmar/fix-pgsql-connect-options
Jobs not running with external PostgreSQL database after PR #6034

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-19 13:55:54 +00:00
Ryan Petrello
06b3e54fb1
remove python2 support from awxkit 2020-03-19 09:02:39 -04:00
Egor Margineanu
f858eda6b1 Made OPTIONS optional 2020-03-19 13:43:06 +01:00
AlanCoding
c5297b0b86
Fix sanity error 2020-03-19 08:42:07 -04:00
Egor Margineanu
e0633c9122 Merge branch 'fix-pgsql-connect-options' of github.com:egmar/awx into fix-pgsql-connect-options 2020-03-19 13:29:39 +01:00
Egor Margineanu
3a208a0be2 Added support for PG port and options. related #6340 2020-03-19 13:29:06 +01:00
Egor Margineanu
cfdfd96793 Added support for PG port and options 2020-03-19 13:26:59 +01:00
Ryan Petrello
db7f0f9421
Merge pull request #6034 from chrismeyersfsu/pg2_no_pubsub
Replace rabbitmq with redis
2020-03-18 17:19:51 -04:00
Ryan Petrello
f1ee963bd0
fix up rebased migrations 2020-03-18 16:19:04 -04:00
Ryan Petrello
7c3cbe6e58
add a license for redis-cli 2020-03-18 16:10:20 -04:00
chris meyers
87de0cf0b3
flake8, pytest, license fixes 2020-03-18 16:10:20 -04:00
chris meyers
18f5dd6e04
add websocket backplane documentation 2020-03-18 16:10:20 -04:00
chris meyers
89163f2915
remove redis broker url test
* We use sockets everywhere. Thus, password special characters no longer
are an issue.
2020-03-18 16:10:20 -04:00
chris meyers
59c9de2761
awxkit python2.7 compatible print
* awxkit still supports python2.7 so do not use fancy f"" yet; instead,
use .format()
2020-03-18 16:10:20 -04:00
chris meyers
b58c71bb74
remove broadcast websockets view 2020-03-18 16:10:20 -04:00
Ryan Petrello
1caa2e0287
work around a limitation in postgres notify to properly support copying
postgres has a limitation on its notify message size (8k), and the
messages we generate for deep copying functionality easily go over this
limit; instead of passing a giant nested data structure across the
message bus, this change makes it so that we temporarily store the JSON
structure in memcached, and look it up from *within* the task

see: https://github.com/ansible/tower/issues/4162
2020-03-18 16:10:20 -04:00
chris meyers
770b457430
redis socket support 2020-03-18 16:10:19 -04:00
chris meyers
d58df0f34a
fix sliding window calculation 2020-03-18 16:10:19 -04:00
chris meyers
3f5e2a3cd3
try to make openshift build happy 2020-03-18 16:10:19 -04:00
chris meyers
2b59af3808
safely operate in async or sync context 2020-03-18 16:10:19 -04:00
chris meyers
9e5fe7f5c6
translate Instance hostname to safe analytics name
* More robust translation of Instance hostname to analytics safe name by
replacing all non-alpha numeric characters with _
2020-03-18 16:10:19 -04:00
chris meyers
093d204d19
fix flake8 2020-03-18 16:10:19 -04:00
chris meyers
e25bd931a1
change dispatcher test to make required queue
* No fallback-default queue anymore. Queue must be explicitly provided.
2020-03-18 16:10:19 -04:00
chris meyers
8350bb3371
robust broadcast websocket error hanndling 2020-03-18 16:10:18 -04:00
chris meyers
d6594ab602
add broadcast websocket metrics
* Gather brroadcast websocket metrics and push them into redis every
configurable seconds.
* Pop metrics from redis in web view layer to display via the api on
demand
2020-03-18 16:10:18 -04:00
chris meyers
b6b9802f9e
increase per-channel capacity
* 100 is the default capacity for a channel. If the client doesn't read
the socket fast enough, websocket messages can and will be lost. This
increases the default to 10,000
2020-03-18 16:10:18 -04:00
chris meyers
0da94ada2b
add missing service name to dev env
* Dev env was bringing the wsbroadcast service up but not under the
tower-process dependency. This is cleaner.
2020-03-18 16:10:18 -04:00
chris meyers
3b9e67ed1b
remove channel group model
* Websocket user session <-> group subscription membership now resides
in Redis rather than the database.
2020-03-18 16:10:18 -04:00
chris meyers
14320bc8e6
handle websocket unsubscribe
* Do not return from blocking unsubscribe until _after_ putting the
gotten unsubscribe message on the queue so that it can be read by the
thread of execution that was unblocked.
2020-03-18 16:10:18 -04:00
chris meyers
3c5c9c6fde
move broadcast websocket out into its own process 2020-03-18 16:10:18 -04:00
chris meyers
f5193e5ea5
resolve rebase errors 2020-03-18 16:10:17 -04:00
chris meyers
03b73027e8
websockets aware of Instance changes
* New tower nodes that are (de)registered in the Instance table are seen
by the websocket layer and connected to or disconnected from by the
websocket broadcast backplane using a polling mechanism.
* This is especially useful for openshift and kubernetes. This will be
useful for standalone Tower in the future when the restarting of Tower
services is not required.
2020-03-18 16:10:17 -04:00
chris meyers
c06b6306ab
remove health info
* Sending health about websockets over websockets is not a great idea.
* I tried sending health data via prometheus and encountered problems
that will need PR's to prometheus_client library to solve. Circle back
to this later.
2020-03-18 16:10:17 -04:00
Shane McDonald
45ce6d794e
Initial migration of rabbitmq -> redis for k8s installs 2020-03-18 16:10:17 -04:00
chris meyers
e94bb44082
replace rabbitmq with redis
* local awx docker-compose and image build only.
2020-03-18 16:10:17 -04:00
chris meyers
be58906aed
remove kombu 2020-03-18 16:10:17 -04:00
chris meyers
403e9bbfb5
add websocket health information 2020-03-18 16:10:16 -04:00
chris meyers
ea29f4b91f
account for isolated job status
* We can not query the dispatcher running on isolated nodes to see if
the playbook is still running because that is the nature of isolated
nodes, they don't run the dispatcher nor do they run the message broker.
Therefore, we should query the control node that is arbitrating the
isolated work. If the control node process in the dispatcher is dead,
consider the iso job dead.
2020-03-18 16:10:16 -04:00
chris meyers
3f2d757f4e
update awxkit to use new unsubscribe event
* Instead of waiting an arbitrary number of seconds. We can now wait the
exact amount of time needed to KNOW that we are unsubscribed. This
changeset takes advantage of the new subscribe reply semantic.
2020-03-18 16:10:16 -04:00
chris meyers
feac93fd24
add websocket group unsubscribe reply
* This change adds more than just an unsubscribe reply.
* Websockets canrequest to join/leave groups. They do so using a single
idempotent request. This change replies to group requests over the
websockets with the diff of the group subscription. i.e. what groups the
user currenntly is in, what groups were left, and what groups were
joined.
2020-03-18 16:10:16 -04:00
chris meyers
088373963b
satisfy generic Role code
* User in channels session is a lazy user class. This does not conform
to what the generic Role ancestry code expects. The Role ancestry code
expects a User objects. This change converts the lazy object into a
proper User object before calling the permission code path.
2020-03-18 16:10:16 -04:00
chris meyers
5818dcc980
prefer simple async -> sync
* asgiref async_to_sync was causing a Redis connection _for each_ call
to emit_channel_notification i.e. every event that the callback receiver
processes. This is a "known" issue
https://github.com/django/channels_redis/pull/130#issuecomment-424274470
and the advise is to slow downn the rate at which you call
async_to_sync. That is not an option for us. Instead, we put the async
group_send call onto the event loop for the current thread and wait for
it to be processed immediately.

The known issue has to do with event loop + socket relationship. Each
connection to redis is achieved via a socket. That conection can only be
waiting on by the event loop that corresponds to the calling thread.
async_to_sync creates a _new thread_ for each invocation. Thus, a new
connection to redis is required. Thus, the excess redis connections that
can be observed via netstat | grep redis | wc -l.
2020-03-18 16:10:16 -04:00
chris meyers
dc6c353ecd
remove support for multi-reader dispatch queue
* Under the new postgres backed notify/listen message queue, this never
actually worked. Without using the database to store state, we can not
provide a at-most-once delivery mechanism w/ multi-readers.
* With this change, work is done ONLY on the node that requested for the
work to be done. Under rabbitmq, the node that was first to get the
message off the queue would do the work; presumably the least busy node.
2020-03-18 16:10:16 -04:00
chris meyers
50b56aa8cb
autobahn 20.1.2 released an hour ago
* 20.1.1 no longer available on pypi
2020-03-18 16:10:15 -04:00