1
0
mirror of https://github.com/ansible/awx.git synced 2024-11-01 16:51:11 +03:00
Commit Graph

504 Commits

Author SHA1 Message Date
Ryan Petrello
7a43f00a5d
add support for HashiCorp signed SSH certificates 2019-04-02 11:20:29 -04:00
Ryan Petrello
4ed5bca5e3
add credential plugin support for Azure Key Vault 2019-04-02 11:20:26 -04:00
Jake McDermott
c209955400
add credential plugin system and minimal working hashivault 2019-04-02 11:19:26 -04:00
Ryan Petrello
563a0cc2a4
move awx.main.expect to awx.main.isolated 2019-03-29 12:14:40 -04:00
Ryan Petrello
495dc2202f
more iso cleanup and bug fixes 2019-03-25 17:47:58 -04:00
Ryan Petrello
af8e071840
remove old callback plugin code and tests 2019-03-25 12:26:51 -04:00
softwarefactory-project-zuul[bot]
8b2fc26219
Merge pull request #3041 from chrismeyersfsu/runnerpy3
ansible-runner integration

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-03-21 22:21:50 +00:00
Matthew Jones
91cbaa1096 Install supervisor into the dev environment 2019-03-21 15:58:24 -04:00
Ryan Petrello
602ef9750f update isolated task execution for ansible-runner 2019-03-18 14:21:47 -04:00
softwarefactory-project-zuul[bot]
c41068edc4
Merge pull request #3420 from baxeno/https_download_postgresql_rpm
docker: yum: use https for postgresql pgdg rpm download

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-03-15 12:04:21 +00:00
Elijah DeLee
d39cfd1778 Update reference to test playbooks 2019-03-14 14:33:16 -04:00
Bruno Thomsen
2e0edcbabd docker: yum: use https for postgresql rpm download.
Signed-off-by: Bruno Thomsen <bruno.thomsen@gmail.com>
2019-03-14 17:14:17 +01:00
Ryan Petrello
50d8eb30e1
fix a failing test in the unit-test target used to generate swagger docs 2019-03-12 14:59:24 -04:00
Shane McDonald
e09684462c Working out some python3 kinks 2019-03-12 11:27:50 -04:00
softwarefactory-project-zuul[bot]
a9226fc25f
Merge pull request #3375 from shanemcd/devel
Fix dev environment when running as root on the host

Reviewed-by: Matthew Jones <mat@matburt.net>
             https://github.com/matburt
2019-03-07 17:20:32 +00:00
Matthew Jones
5cf7cc21c8
Revert "Fix chrome can not be started with unit-tests due to missing shared libraries"
This reverts commit d558ffd699.
2019-03-07 10:07:08 -05:00
Matthew Jones
499fd7b2f1
Revert "Fix chrome can not be started with unit-tests due to missing shared libraries"
This reverts commit 3e5f328b52.
2019-03-07 10:06:30 -05:00
Shane McDonald
b29a9cd86e Fix dev environment when running as root on the host
Without this, CURRENT_UID isnt actually passed in from the host, and wipes out /etc/passwd even when we’re actually running as root.

I tested this as a non-root user on Linux, and on Docker for Mac
2019-03-06 17:08:56 -05:00
softwarefactory-project-zuul[bot]
658e5f0fc8
Merge pull request #3272 from jladdjr/mo_stats
Add support for new playbook stats

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-02-28 18:24:03 +00:00
Jim Ladd
b7b0bdaeca Ansible 2.8 deprecates use of -U 2019-02-25 00:42:19 -08:00
Kia Lam
d2c345a374
Fix API lint error. 2019-02-22 13:06:20 -05:00
Kia Lam
0943f989ce
Fix various errors when trying to run make bulk_data.
- Properly import PrimoridialModel.
- Use floor division operator for range() method to avoid float vs int errors.
2019-02-22 11:59:18 -05:00
softwarefactory-project-zuul[bot]
889dae357b
Merge pull request #3235 from ryanpetrello/sql-profiling
add a custom DB backend that provides system-level SQL profiling

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-02-15 21:56:28 +00:00
Ryan Petrello
6f43875e80
record profile data in /var/log/tower, not /var/lib/awx 2019-02-15 14:34:55 -05:00
Ryan Petrello
eed94b641e
add a custom DB backend that provides system-level SQL profiling
run this command on _any_ node in an awx cluster:

$ awx-manage profile_sql --threshold=2.0 --minutes=1

...and for 1 minute, the timing for _every_ SQL query in _every_ awx
Python process that uses the Django ORM will be measured

queries that run longer than (in this example) 2 seconds will be
written to a per-process sqlite database in /var/lib/awx/profile, and
the file will contain an EXPLAIN VERBOSE for the query and the full
Python stack that led to that SQL query's execution (this includes not
just WSGI requests, but background processes like the runworker and
dispatcher)

$ awx-manage profile_sql --threshold=0

...can be used to disable profiling again (if you don't want to wait for
the minute to expire)
2019-02-14 15:04:46 -05:00
Ryan Petrello
9bebf3217e
remove usage of import * and enforce F405 in our linter
import * is a scourge upon the earth
2019-02-13 17:10:33 -05:00
Hideki Saito
3e5f328b52 Fix chrome can not be started with unit-tests due to missing shared libraries
- Added installation packages conforming to the unit-tests/Dockerfile

Signed-off-by: Hideki Saito <saito@fgrep.org>
2019-01-31 13:02:23 +09:00
Hideki Saito
d558ffd699 Fix chrome can not be started with unit-tests due to missing shared libraries
- Modify Dockerfile to install necesarry shared libraries for chrome

Signed-off-by: Hideki Saito <saito@fgrep.org>
2019-01-31 09:54:55 +09:00
Jake McDermott
73f617d811
Install dependencies for Chromium in unit test image 2019-01-28 15:57:17 -05:00
Ryan Petrello
daeeaf413a
clean up unnecessary usage of the six library (awx only supports py3) 2019-01-25 00:19:48 -05:00
Shane McDonald
d273472927
Enable py3 SCL if needed 2019-01-15 14:09:05 -05:00
Author: Jim Ladd
2c5bdf3611
fix some isolated py3 bugs 2019-01-15 14:09:05 -05:00
Ryan Petrello
c6482137d1
parametrize PYTHON for Ubuntu py35 support 2019-01-15 14:09:05 -05:00
Ryan Petrello
f223df303f
convert py2 -> py3 2019-01-15 14:09:01 -05:00
Ryan Petrello
f132ce9b64 switch image builds to py3 2019-01-15 13:25:13 -05:00
AlanCoding
77d2364022
Make docker environment interoperable with supervisorctl commands 2019-01-10 13:41:15 -05:00
Ryan Petrello
a810aaf319
fix isolated nodes in the dev environment 2018-11-28 09:54:39 -05:00
AlanCoding
0b32733dc8
set fixed container names 2018-11-26 08:26:57 -05:00
Ryan Petrello
b4f906ceb1
add dispatcher status to the sosreport 2018-11-20 12:12:02 -05:00
Ryan Petrello
a748a272fb
Merge remote-tracking branch 'tower/release_3.3.1' into devel 2018-11-01 12:07:02 -04:00
AlanCoding
19030b9d5f
apply docker-compose fix to cluster target too 2018-10-26 09:36:11 -04:00
Shane McDonald
a361b5da6e
Fix permissions when running dev container as non-root user
I wanted to pass `—user` to `docker-compose` up, but that option doesnt exist. To get around this, I had to record the uid on the host (CURRENT_UID), interpolate the variable in tools/docker-compose.yml, and detect that inside the container. I then piggy-backed on the /etc/passwd hack we use for scenarios with unpredictable uids.
2018-10-24 10:30:04 -04:00
Shane McDonald
7df63830ed
Remove reference to file that doesnt exist anymore 2018-10-24 10:30:03 -04:00
Matthew Jones
119b9475ea
Force openshift user behavior for uids over 2500 2018-10-16 11:30:04 -04:00
Hideki Saito
d5626a4f3e [3.3.1] Add files and output of commands to gather with sosreport
- Fixed issue #3064
2018-10-15 11:40:51 +09:00
softwarefactory-project-zuul[bot]
0a964b2bf6
Merge pull request #2266 from ansible/celery-tastes-bad
replace the celery-based task queue with a kombu-based implementation

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2018-10-12 18:40:54 +00:00
Ryan Petrello
ac80bc874a
more shippable -> zuul cleanup 2018-10-12 11:50:29 -04:00
Ryan Petrello
ff1e8cc356
replace celery task decorators with a kombu-based publisher
this commit implements the bulk of `awx-manage run_dispatcher`, a new
command that binds to RabbitMQ via kombu and balances messages across
a pool of workers that are similar to celeryd workers in spirit.
Specifically, this includes:

- a new decorator, `awx.main.dispatch.task`, which can be used to
  decorate functions or classes so that they can be designated as
  "Tasks"
- support for fanout/broadcast tasks (at this point in time, only
  `conf.Setting` memcached flushes use this functionality)
- support for job reaping
- support for success/failure hooks for job runs (i.e.,
  `handle_work_success` and `handle_work_error`)
- support for auto scaling worker pool that scale processes up and down
  on demand
- minimal support for RPC, such as status checks and pool recycle/reload
2018-10-11 10:53:30 -04:00
Ryan Petrello
f87a09c46a
build swagger docs as part of CI 2018-10-10 10:27:54 -04:00
Shane McDonald
7a5cfd05a3 Run tests in Docker as non-root user 2018-10-09 15:16:01 -04:00
Ryan Petrello
ee11341430
more make clean tinkering 2018-10-05 13:40:08 -04:00
Matthew Jones
56263a5fea
Force ui cleanup in the test environment
Also allow using the system make
2018-10-05 13:40:07 -04:00
Matthew Jones
3a8bacb8ef
Add an initial check and gate job configuration for zuul
Updates for running ui tests and linters
2018-10-05 13:39:59 -04:00
Shane McDonald
1b4c3f56fa
Merge pull request #2113 from kialam/upgrade-node-lts
Upgrade Node and NPM to LTS
2018-09-18 12:46:30 -04:00
Shane McDonald
1371e394de Update Node version in dev container image 2018-09-18 12:37:20 -04:00
Shane McDonald
580004b395 Merge remote-tracking branch 'downstream/release_3.3.0' into devel
# Conflicts:
#	awx/main/notifications/slack_backend.py
2018-09-18 08:11:50 -04:00
Shane McDonald
07cb2aa9bb Fix / improve minishift dev env playbook
- Redo how we detect / set the minishift path
- Log into the correct admin account, once.
- Make sure commands that fail cause tasks to fail (s/;/&&/)
2018-08-15 15:01:40 -04:00
Ryan Petrello
18f6f68540
Merge remote-tracking branch 'tower/release_3.3.0' into devel 2018-08-10 11:54:34 -04:00
Ryan Petrello
6f5259d017
remove the network UI 2018-07-30 11:03:53 -04:00
Christian Adams
95a32aaa6e
Merge pull request #2606 from rooftopcellist/update_logging_doc
Update logging doc
2018-07-22 16:15:28 -04:00
chris meyers
7d24566120 more robustness 2018-07-20 15:02:44 -04:00
adamscmRH
ef29dab013 update logging readme 2018-07-19 11:41:39 -04:00
Matthew Jones
190525e835 Adding cluster development service definitions 2018-07-19 10:39:08 -04:00
Matthew Jones
5c400cdf79 Add local minishift development tooling
Based on mapping the local development tree through minishift
hostfolder interface.
2018-07-19 10:39:08 -04:00
Rob Ruma
6378479ee7 Updates to callback scripts that include retry functionality for bash and extra_vars handling for PowerShell
Signed-off-by: Rob Ruma <robruma@users.noreply.github.com>
2018-07-09 13:00:10 -04:00
Rob Ruma
6514c338a2 Adding ability to ignore SSL certificate validation errors, cleanup parameters and usage
Signed-off-by: Rob Ruma <robruma@users.noreply.github.com>
2018-07-06 23:18:12 -04:00
Bill Nottingham
0d03036fb1
Merge pull request #1629 from robruma/feature/update_callback_scripts
Updates to callback scripts
2018-06-25 15:47:25 -04:00
Guoqiang Zhang
530860c386 Force the python IO encoding to be utf_8 when using honcho 2018-06-22 09:35:28 -04:00
Ryan Petrello
84eacfc360
fix a few isolated dev issues
the main goal of this change is to make `make docker-isolated` work out
of the box

- specify the proper version for awx-expect --version
- update some deprecated playbook bits
- change the isolated container to privileged so bwrap will work
- fix awx-manage test_isolated_connection
- expedite the first isolated heartbeat so you don't have to wait 10m;
  this is accomplished by _not_ setting Instance.last_isolated_check to
  now() at insertion time (which causes the next check not to happen for
  10 minutes)
- fix a bug that caused isolated node execution to fail when bwrap was
  enabled

see: https://github.com/ansible/tower/issues/2150

This reverts commit 9863fe71dc.
2018-06-13 14:17:58 -04:00
Ryan Petrello
1733a20094
make sdb-listen work for docker-compose-cluster
use a different port range for each container, because docker can't map
them all to the same port range
2018-06-12 13:57:28 -04:00
Matthew Jones
22dd6ddfea
Remove the logstash container from the base dev docker compose
Now with less java running while you code!
2018-06-05 09:42:38 -04:00
chris meyers
9863fe71dc do not require privileged iso container
* The init call w/ privileged was causing my laptop to wig out. This
changeset still functions w/ out requiring privileged access.
2018-06-04 10:06:59 -04:00
Ryan Petrello
3abdf66794 run network ui tests in shippable and Jenkins 2018-05-24 09:07:53 -04:00
Ben Thomasson
89cabf7ca0
Fixes timeout when exporting YAML from network UI
Exporting YAML on dev envs with honcho and in production environments
would timeout.  This was due to daphne handling the export request
in dev but not in production.  This fixes network_ui to use uwsgi instead
of daphne to handle the request.
2018-05-02 10:31:36 -04:00
Wayne Witzel III
7151071779 Add awx-link make target 2018-04-25 11:55:57 -04:00
AlanCoding
4036f64cc4
update the dev supervisor file to match recent change 2018-04-20 14:53:21 -04:00
Alan Rominger
c0fd56c654
Merge pull request #1381 from AlanCoding/test_docker
Run ansible tests separately
2018-04-19 13:04:14 -04:00
Bill Nottingham
ea9278a3ea Drop /var/lib/awx/job_status listing from sosreport.
This directory is not persistently populated any more.
2018-04-19 10:47:45 -04:00
Bill Nottingham
40cf80db1d Add rabbitmq status commands as well. 2018-04-17 12:25:26 -04:00
Bill Nottingham
ade51c93d8 Add some more information to sosreports.
- tower cluster configuration
- list of all venvs (including custom ones)
2018-04-17 12:25:26 -04:00
AlanCoding
5eaffb3520
run ansible tests separately 2018-04-16 10:32:51 -04:00
Ben Thomasson
701150bd1a
Adds configuration for the network-ui websocket
* Configures NGINX for the network-ui websocket.
* Configures supervisor.conf for network_ui websocket.
2018-03-23 17:00:14 -04:00
Ben Thomasson
48d801271c
Imports prototype from ansible-network-ui
The ansible-network-ui prototype project builds a standalone Network UI
outside of Tower as its own Django application. The original prototype
code is located here:
https://github.com/benthomasson/ansible-network-ui.

The prototype provides a virtual canvas that supports placing
networking devices onto 2D plane and connecting those devices together
with connections called links.  The point where the link connects
to the network device is called an interface.  The devices, interfaces,
and links may all have their respective names.  This models physical
networking devices is a simple fashion.

The prototype implements a pannable and zoomable 2D canvas in using SVG
elements and AngularJS directives.   This is done by adding event
listeners for mouse and keyboard events to an SVG element that fills the
entire browser window.

Mouse and keyboard events are handled in a processing pipeline where
the processing units are implemented as finite state machines that
provide deterministic behavior to the UI.

The finite state machines are built in a visual way that makes
the states and transitions clearly evident.  The visual tool for
building FSM is located here:
https://github.com/benthomasson/fsm-designer-svg.   This tool
is a fork of this project where the canvas is the same.  The elements
on the page are FSM states and the directional connections are called
transitions.   The bootstrapping of the FSM designer tool and
network-ui happen in parallel.  It was useful to try experiemental
code in FSM designer and then import it into network-ui.

The FSM designer tool provides a YAML description of the design
which can be used to generate skeleton code and check the implementation
against the design for discrepancies.

Events supported:

* Mouse click
* Mouse scroll-wheel
* Keyboard events
* Touch events

Interactions supported:

* Pan canvas by clicking-and-dragging on the background
* Zooming canvas by scrolling mousewheel
* Adding devices and links by using hotkeys
* Selecting devices, interaces, and links by clicking on their icon
* Editing labels on devices, interfaces, and links by double-clicking on
  their icon
* Moving devices around the canvas by clicking-and-dragging on their
  icon

Device types supported:

* router
* switch
* host
* racks

The database schema for the prototype is also developed with a visual
tool that makes the relationships in the snowflake schema for the models
quickly evident.  This tool makes it very easy to build queries across
multiple tables using Django's query builder.

See: https://github.com/benthomasson/db-designer-svg

The client and the server communicate asynchronously over a websocket.
This allows the UI to be very responsive to user interaction since
the full request/response cycle is not needed for every user
interaction.

The server provides persistence of the UI state in the database
using event handlers for events generated in the UI.  The UI
processes mouse and keyboard events, updates the UI, and
generates new types of events that are then sent to the server
to be persisted in the database.

UI elements are tracked by unique ids generated on the client
when an element is first created.  This allows the elements to
be correctly tracked before they are stored in the database.

The history of the UI is stored in the TopologyHistory model
which is useful for tracking which client made which change
and is useful for implementing undo/redo.

Each message is given a unique id per client and has
a known message type.  Message types are pre-populated
in the MessageType model using a database migration.

A History message containing all the change messages for a topology is
sent when the websocket is connected.  This allows for undo/redo work
across sessions.

This prototype provides a server-side test runner for driving
tests in the user interface.  Events are emitted on the server
to drive the UI.  Test code coverage is measured using the
istanbul library which produces instrumented client code.
Code coverage for the server is is measured by the coverage library.

The test code coverage for the Python code is 100%.
2018-03-23 17:00:14 -04:00
Rob Ruma
e8b026648f Updates to callback scripts that include retry functionality for bash and extra_vars handling for PowerShell
Signed-off-by: Rob Ruma <robruma@users.noreply.github.com>
2018-03-21 08:22:08 -04:00
adamscmRH
8d460490c1 add xmlsec flag to docker installs 2018-03-14 14:28:35 -04:00
Ryan Petrello
13672cc88c
fix busted shippable builds 2018-03-05 14:16:42 -05:00
chris meyers
b69315f2eb fix up the config map watcher script
* invoke main() in config watcher script
* correctly call hash update by passing the filename
2018-03-01 17:06:07 -05:00
Ryan Petrello
d743b77353 replace our rdb tooling w/ the sdb PyPI package 2018-02-26 19:05:50 -05:00
Bill Nottingham
c1203942e0 Remove obsolete ansible_awx.egg-info. 2018-02-26 15:04:37 -05:00
Chris Church
4ec683efcb
Fix dummy data generator for WFJT node credentials. 2018-02-21 08:55:43 -05:00
Matthew Jones
0d2daecf49
Merge pull request #1243 from matburt/fix_clustering_isolated
Fix isolated instance clustering implementation
2018-02-14 08:32:24 -05:00
Matthew Jones
925d9efecf
Fixing up isolated node execution after cluster changes
* Rework queue detection to include control groups and isolated instances
* Fix up development tooling around isolated nodes
* Update unit tests
2018-02-13 21:51:38 -05:00
cclauss
260aec543e Misc Python 3 changes: execfile(), file(), reduce(), StandardError 2018-02-09 17:17:05 +01:00
Alan Rominger
b3dcfc8c18
Merge pull request #903 from ansible/item_copy
Implement item copy feature
2018-02-08 15:51:16 -05:00
cclauss
e18838a4b7 Modernize Python 2 code to get ready for Python 3 2018-02-08 17:26:22 +01:00
Matthew Jones
c819560d39
Add automatic deprovisioning support, only enabled for openshift
* Implement a config watcher for service restarts
* If the configmap bind point changes then restart all services
2018-02-01 16:51:40 -05:00
Chris Meyers
0e97dc4b84
Beat and celery clustering fixes
* use embedded beat rather than standalone
* dynamically set celeryd hostname at runtime
* add embeded beat flag to celery startup
* Embedded beat mode routes will piggyback off of celery worker setup
signal
2018-02-01 16:47:33 -05:00
Chris Meyers
c9ff3e99b8
celeryd attach to queues dynamically
* Based on the tower topology (Instance and InstanceGroup
relationships), have celery dyamically listen to queues on boot
* Add celery task capable of "refreshing" what queues each celeryd
worker listens to. This will be used to support changes in the topology.
* Cleaned up some celery task definitions.
* Converged wrongly targeted job launch/finish messages to 'tower'
queue, rather than a 1-off queue.
* Dynamically route celery tasks destined for the local node
* separate beat process

add support for separate beat process
2018-02-01 16:37:33 -05:00
Shane McDonald
67c6591f6f Use newer version of git in dev image
More fallout from #982
2018-01-22 13:57:44 -05:00
Aaron Tan
a2fd78add4 Implement item copy feature
See acceptance doc for implement details.

Signed-off-by: Aaron Tan <jangsutsr@gmail.com>
2018-01-02 10:20:44 -05:00