IF YOU WOULD LIKE TO GET AN ACCOUNT, please write an
email to Administrator. User accounts are meant only to access repo
and report issues and/or generate pull requests.
This is a purpose-specific Git hosting for
BaseALT
projects. Thank you for your understanding!
Только зарегистрированные пользователи имеют доступ к сервису!
Для получения аккаунта, обратитесь к администратору.
this commit implements the bulk of `awx-manage run_dispatcher`, a new
command that binds to RabbitMQ via kombu and balances messages across
a pool of workers that are similar to celeryd workers in spirit.
Specifically, this includes:
- a new decorator, `awx.main.dispatch.task`, which can be used to
decorate functions or classes so that they can be designated as
"Tasks"
- support for fanout/broadcast tasks (at this point in time, only
`conf.Setting` memcached flushes use this functionality)
- support for job reaping
- support for success/failure hooks for job runs (i.e.,
`handle_work_success` and `handle_work_error`)
- support for auto scaling worker pool that scale processes up and down
on demand
- minimal support for RPC, such as status checks and pool recycle/reload
This was causing offline pip installs to fail for some weird reason:
The 'setuptools_scm>=1.15.0' distribution was not found and is required by the application
Even though it is there. v2.x still works.
The ansible-network-ui prototype project builds a standalone Network UI
outside of Tower as its own Django application. The original prototype
code is located here:
https://github.com/benthomasson/ansible-network-ui.
The prototype provides a virtual canvas that supports placing
networking devices onto 2D plane and connecting those devices together
with connections called links. The point where the link connects
to the network device is called an interface. The devices, interfaces,
and links may all have their respective names. This models physical
networking devices is a simple fashion.
The prototype implements a pannable and zoomable 2D canvas in using SVG
elements and AngularJS directives. This is done by adding event
listeners for mouse and keyboard events to an SVG element that fills the
entire browser window.
Mouse and keyboard events are handled in a processing pipeline where
the processing units are implemented as finite state machines that
provide deterministic behavior to the UI.
The finite state machines are built in a visual way that makes
the states and transitions clearly evident. The visual tool for
building FSM is located here:
https://github.com/benthomasson/fsm-designer-svg. This tool
is a fork of this project where the canvas is the same. The elements
on the page are FSM states and the directional connections are called
transitions. The bootstrapping of the FSM designer tool and
network-ui happen in parallel. It was useful to try experiemental
code in FSM designer and then import it into network-ui.
The FSM designer tool provides a YAML description of the design
which can be used to generate skeleton code and check the implementation
against the design for discrepancies.
Events supported:
* Mouse click
* Mouse scroll-wheel
* Keyboard events
* Touch events
Interactions supported:
* Pan canvas by clicking-and-dragging on the background
* Zooming canvas by scrolling mousewheel
* Adding devices and links by using hotkeys
* Selecting devices, interaces, and links by clicking on their icon
* Editing labels on devices, interfaces, and links by double-clicking on
their icon
* Moving devices around the canvas by clicking-and-dragging on their
icon
Device types supported:
* router
* switch
* host
* racks
The database schema for the prototype is also developed with a visual
tool that makes the relationships in the snowflake schema for the models
quickly evident. This tool makes it very easy to build queries across
multiple tables using Django's query builder.
See: https://github.com/benthomasson/db-designer-svg
The client and the server communicate asynchronously over a websocket.
This allows the UI to be very responsive to user interaction since
the full request/response cycle is not needed for every user
interaction.
The server provides persistence of the UI state in the database
using event handlers for events generated in the UI. The UI
processes mouse and keyboard events, updates the UI, and
generates new types of events that are then sent to the server
to be persisted in the database.
UI elements are tracked by unique ids generated on the client
when an element is first created. This allows the elements to
be correctly tracked before they are stored in the database.
The history of the UI is stored in the TopologyHistory model
which is useful for tracking which client made which change
and is useful for implementing undo/redo.
Each message is given a unique id per client and has
a known message type. Message types are pre-populated
in the MessageType model using a database migration.
A History message containing all the change messages for a topology is
sent when the websocket is connected. This allows for undo/redo work
across sessions.
This prototype provides a server-side test runner for driving
tests in the user interface. Events are emitted on the server
to drive the UI. Test code coverage is measured using the
istanbul library which produces instrumented client code.
Code coverage for the server is is measured by the coverage library.
The test code coverage for the Python code is 100%.
Last round of dependency updates showed that AWX
depended on packages which came implicitly from shade
decorator is added as an explicit dependency
and all of the rest of shade requirements are
added back in here.
Upgrades of minor dependency upgrades
Inventory scripts were upgraded in separate commit
Major exclusions from this update
- celery was already downgraded for other reasons
- Django / DRF major update already done, minor bumps here
- asgi-amqp has fixes coming independently, not touched
- TACACS plus added features not needed
Removals of note
- remove shade from AWX requirements
- remove kombu from Ansible requirements
Other notes
Add note about pinning setuptools and pip,
done but not mentioned previously
Stop pinning gevent-websocket and twisted
upgrade Azure to Ansible core requirements
more detailed notes
https://gist.github.com/AlanCoding/9442a512ab6977940bc7b5b346d4f70b
upgrade version of Django for Exception
update our event data search algorithm to be a bit lazier in event data
discovery; this drastically improves processing speeds for stdout >5MB
see: https://github.com/ansible/awx/issues/417
* Jupyter starts alongside the other awx services and is available on
0.0.0.0:8888
* make target: make jupyter
* default settings in settings/development.py
* Added jupyter, matplotlib, numpy to dev dependencies
We `pip download` this file for offline installs. Automat lists this package as a setup_requires, but `pip download` doesn’t resolve these dependencies (distutils will attempt to install them via easy_install when setup.py is invoked).
* colorize uwsgi and celery logs; DEBUG lines are green, WARN lines
are yellow, ERROR lines (and tracebacks) are red
* pretty-print fact callback receiver JSON
* simplify the uwsgi log format so it's more legible
instead of launching isolated tasks via `systemctl`, treat
`awx.main.isolated.run` as an executable that knows how to daemonize
additionally, add `setup.py isolated_build` for isolated Tower source
distribution
* Dynamic Inventory Source
Template against ansible 2.3 dynamic inventory sources.
The major change is removal of `rax.py`. Most upstream scripts except
`foreman.py` has quite trivial coding style changes, or minor functional
extensions that does not affect Tower inventory update runs.
`foreman.py`, on the other hand, went through quite a major refactoring,
but functionalities stay the same.
Major python dependency updates include apache-libcloud (1.3.0 -->
2.0.0), boto (2.45.0 --> 2.46.1) and shade (1.19.0 --> 1.20.0). Minor
python dependency updates include indirect updates via `pip-compile`,
which are determined by base dependencies.
Some minor `task.py` extensions:
- `.ini` file for ec2 has one more field `stack_filter=False`, which
reveals changes in `ec2.py`.
- `.ini` file for cloudforms will catch these four options from
`source_vars_dict` of inventory update: `'version', 'purge_actions',
'clean_group_keys', 'nest_tags'`. These four options have always been
available in `cloudforms.py` but `cloudforms.ini.example` has not
mentioned them until the latest version. For consistency with upstream
docs, we should make these fields available for tower user to customize.
- YAML file of openstack will catch ansible options `use_hostnames`,
`expand_hostvars` and `fail_on_errors` from `source_vars_dict` of
inventory update as a response to issue #6075.
* Remove Rackspace support
Supports of Rackspace as both a dynamic inventory source and a cloud
credential are fully removed. Data migrations have been added to support
arbitrary credential types feature and delete rackspace inventory
sources.
Note also requirement `jsonschema` has been moved from
`requirements.txt` to `requirements.in` as a primary dependency to
reflect it's usage in `/main/fields.py`.
Connected issue: #6080.
* `pexpect` major update
`pexpect` stands at the very core of our task system and underwent a
major update from 3.1 to 4.2.1. Although verified during devel, please
still be mindful of any suspicious issues on celery side even after this
PR gets merged.
* Miscellaneous
- requests now explicitly declared in `requirements.in` at version 2.11.1
in response to upstream issue
- celery: 3.1.17 -> 3.1.25
- django-extensions: 1.7.4 -> 1.7.8
- django-polymorphic: 0.7.2 -> 1.2
- django-split-settings: 0.2.2 -> 0.2.5
- django-taggit: 0.21.3 -> 0.22.1
- irc: 15.0.4 -> 15.1.1
- pygerduty: 0.35.1 -> 0.35.2
- pyOpenSSL: 16.2.0 -> 17.0.0
- python-saml: 2.2.0 -> 2.2.1
- redbaron: 0.6.2 -> 0.6.3
- slackclient: 1.0.2 -> 1.0.5
- tacacs_plus: 0.1 -> 0.2
- xmltodict: 0.10.2 -> 0.11.0
- pip: 8.1.2 -> 9.0.1
- setuptools: 23.0.0 -> 35.0.2
- (requirements_ansible.in only)kombu: 3.0.35 -> 3.0.37
I had to pull the git urls out of the main requirements files because in order to install offline (--no-index), we need pip to install from local package archives rather than cloning repo.
The weird `cat` thing going on in the Makefile is because we need to install everything as part of a single `pip install` transaction. Without this, installing only requirements_git.txt will result in dependencies getting unintentionally updated.
These packages were getting silently updated, making vendoring these dependencies difficult.
This ensures that the expected versions of these packages are installed in the virtualenvs.
event emitter now caches to memcache where it is eventually picked up
by the stdout event emitter. This obviates event reassembly in the
callback receiver.
* We have the requirements that users with Tower < 3.0.x must upgrade to
3.0.x before > 3.0.x. Thus, we are save to delete all mongo migration
code and deps.
* Set --master flag so we have a preforking master process to manage
subprocs
* Set max-requests to 1000 so uwsgi will recycle processes after 1000
requests
* Set --no-orphans so uwsgi will track child processes and clean them up
in order to respawn children
* Turn stats on in the container and install uwsgitop in dev
requirements so we can track uwsgi stats
* release_3.0.3: (55 commits)
Revert "Revert "Add needed types for selinux change""
Revert "Add needed types for selinux change"
interpret backslash escapes when displaying url in welcome message
Bump the SELinux policy version
Add needed types for selinux change
Update SELinux policy to allow httpd_t to execute files in lib_t and var_lib_t
Bumping changelog for 3.0.3
Update rax.py inventory
Revert "filter internal User.admin_roles from the /roles API list view"
fix spelling of disassociated
Resolves 404 when assigning resources/users to organizations in card view. Sidesteps a bug in the Refresh() utility, where pagination calculations are not made against filtered results.
Sync azure changes to Tower virtual environment
Add regions here as well.
Also bump boto for new regions, per ryansb.
More regions!
Revert "bump shade version"
bump shade version
Hack copying of job_template.related.survey_spec into ui job copy flow, resolves#3737
Revert "bump shade version"
bump shade version
...
- current Ansible stuff (2.1.1) requires azure SDK 2.0.0rc5
- requests 2.10.0 is the published minimum version for pywinrm, as it contains a bugfix for catastrophic SSL tunnel failure on large payloads that pywinrm hits frequently, 2.11.0 is best tested.
* Add separate Django app for configuration: awx.conf.
* Migrate from existing main.TowerSettings model to conf.Setting.
* Add settings wrapper to allow get/set/del via django.conf.settings.
* Update existing references to tower_settings to use django.conf.settings.
* Add a settings registry to allow for each Django app to register configurable settings.
* Support setting validation and conversion using Django REST Framework fields.
* Add /api/v1/settings/ to display a list of setting categories.
* Add /api/v1/settings/<slug>/ to display all settings in a category as a single object.
* Allow PUT/PATCH to update setting singleton, DELETE to reset to defaults.
* Add "all" category to display all settings across categories.
* Add "changed" category to display only settings configured in the database.
* Support per-user settings via "user" category (/api/v1/settings/user/).
* Support defaults for user settings via "user-defaults" category (/api/v1/settings/user-defaults/).
* Update serializer metadata to support category, category_slug and placeholder on OPTIONS responses.
* Update serializer metadata to handle child fields of a list/dict.
* Hide raw data form in browsable API for OPTIONS and DELETE.
* Combine existing licensing code into single "TaskEnhancer" class.
* Move license helper functions from awx.api.license into awx.conf.license.
* Update /api/v1/config/ to read/verify/update license using TaskEnhancer and settings wrapper.
* Add support for caching settings accessed via settings wrapper.
* Invalidate cached settings when Setting model changes or is deleted.
* Preload all database settings into cache on first access via settings wrapper.
* Add support for read-only settings than can update their value depending on other settings.
* Use setting_changed signal whenever a setting changes.
* Register configurable authentication, jobs, system and ui settings.
* Register configurable LDAP, RADIUS and social auth settings.
* Add custom fields and validators for URL, LDAP, RADIUS and social auth settings.
* Rewrite existing validator for Credential ssh_private_key to support validating private keys, certs or combinations of both.
* Get all unit/functional tests working with above changes.
* Add "migrate_to_database_settings" command to determine settings to be migrated into the database and comment them out when set in Python settings files.
* Add support for migrating license key from file to database.
* Remove database-configuable settings from local_settings.py example files.
* Update setup role to no longer install files for database-configurable settings.
f 94ff6ee More settings work.
f af4c4e0 Even more db settings stuff.
f 96ea9c0 More settings, attempt at singleton serializer for settings.
f 937c760 More work on singleton/category views in API, add code to comment out settings in Python files, work on command to migrate settings to database.
f 425b0d3 Minor fixes for sprint demo.
f ea402a4 Add support for read-only settings, cleanup license engine, get license support working with DB settings.
f ec289e4 Rename migration, minor fixmes, update setup role.
f 603640b Rewrite key/cert validator, finish adding social auth fields, hook up signals for setting_changed, use None to imply a setting is not set.
f 67d1b5a Get functional/unit tests passing.
f 2919b62 Flake8 fixes.
f e62f421 Add redbaron to requirements, get file to database migration working (except for license).
f c564508 Add support for migrating license file.
f 982f767 Add support for regex in social map fields.
* ramparts_and_ha: (21 commits)
Rename database migrations for devel integration
Integrate callback receiver refactoring
Fix an issue running jobs in the cluster
Implement a more dynamic celery queue system
Purge old munin monitors and tools
Refactor Tower HA Instance logic and models
Docker compose improvements
Initial Docker Compose workflow for Tower cluster
Add memcached role for setup playbook
Removing qpid from deb packaging
Refactor rabbitmq role
Integrate memcached into setup playbook
Remove mongodb shutdown task
Remove dependency on erlang_sd_notify
Add initial rabbitmq role
Initial rabbitmq setup playbook integration
Update development environment for rabbit
Replace qpid with rabbitmq
Remove redis role from setup playbook
Update qpid packaging, remove migrations
...
* Drop ZMQ as the communication mechanism between job_event_callback and
callback_receiver
* Setup queue and exchange for callback broker communication
* Refactor event plugin and callback receiver to efficiently handle
message submission and processing
* Integrate django caching for parent processing
* Meant to be a starting point to more efficiently manage work routing
and to balance work across all tower nodes
* Integrate flower as a dev tool that starts alongside other nodes.
Helpful for observing and monitoring the queues/exchanges
* For the moment, force the task manager to only run on one node (not
sure if this is needed)
* Define queues and routes for all task work
* Bump celery version to 3.1.23
* Expose flower through haproxy
* ramparts_and_ha:
Removing qpid from deb packaging
Refactor rabbitmq role
Integrate memcached into setup playbook
Remove mongodb shutdown task
Remove dependency on erlang_sd_notify
Add initial rabbitmq role
Initial rabbitmq setup playbook integration
Update development environment for rabbit
Replace qpid with rabbitmq
Remove redis role from setup playbook
Update qpid packaging, remove migrations
Integrate packaging for qpid/memcached
* Switch base tower devel image from u14.04 to c7
* Switch container image to build python dependencies into itself
instead of forcing it to be built on startup
* Upgrade venv pip to 8.1.2
* Neuter queue.py which was heavily tied to redis and was basically
orphaned code
* Alter local_settings to override default cache settings and use
memcached
* Alter local settings to refer to qpid instead of redis for celery
broker
* Purge redis python dependencies and add qpid and memcached
* Update docker-compose to purge redis and add qpid and memcached
- current Ansible stuff (2.1.1) requires azure SDK 2.0.0rc5
- requests 2.10.0 is the published minimum version for pywinrm, as it contains a bugfix for catastrophic SSL tunnel failure on large payloads that pywinrm hits frequently, 2.11.0 is best tested.
* release_2.4.5:
Remove distribute from the setup virtualenv if installed (#1631)
Update changelogs for 2.4.5 release
Only export changed targets in reprepro
Improve the efficiency of the stdout dump database migration
Added logic to not show the loop summary events in the UI by looking at event_data.event_loop.
Handle runner items from ansible v2
bump boto
fix case of Ansible v2 _result.cmd is list for release 2.4.5
Handle both string and list hosts in our hosts->name conversion
Default play names to the hosts the play was run against
Use better isinstance(x) type checking
Backporting test fixes from PR #1020: Fix error with ad hoc command events when running in check mode.
Fix for tasks breaking when using 'yum' with ansible 1.9.4
Bump 2.4.5 version, changelogs, and reprepo
* Vendor ansible's azure_rm inventory script
* Add new inventory type
* Add new credential type
* Expand host instance_id column from varchar 100 to 1024 to accept the
long instance ids returned by Azure
* Make the inventory_import azure match rename more explicit.
* Break requirements down into ansible and tower reqs
* Generate separate Ansible and Tower virtual environments
* Install appropriate requirements files into each one
* Modify development tools to use these venvs instead of our old
site-packages
* Modify settings to indicate venv enablement and location of venvs
* Modify tasks to use the proper virtual environment for its purpose
* flake8 monkeypatches pyflakes to add codes to linting results. It
would seem that pyflakes 1.1.0 adds more checks but flake8 doesn't
monkeypatch to expose them. I've submitted a PR to add error code 405 to
be able to express that we ignore it in our setup.cfg
https://gitlab.com/pycqa/flake8/merge_requests/56
* notifications_work: (23 commits)
Updates to notification unit tests after @wwitzel3's feedback
Fix some notifications issues and write some tests
Add notification system documentation
Clean up flake8 related issues
Fixing up some unicode issues
Implement tower ui view url on models
Sanity check and force proper types in admin check
Proper type for in check
Adding migration and base notification type
Add a periodic administrative notification
Refactor message generator
Support notification password field encryption
Notification configuration type checking
Refactor NotificationTemplate to Notifier
Implement irc notification backend
Add webhook notification backend
Pagerduty and Hipchat backends plus some cleanup
Notification serializers, views, and tasks
Implement notification serializer and validations
Notification endpoints and url expositions
...
* awx.main.models Fact added
* view host fact and timeline updated to use new Postgres Fact model instead of Mongo
* Removed license set start Mongo logic
* added View tests
* added Model tests
* Removed mongo fact unit tests
* point at modified jsonbfield that supports sqlite storage driver
* postgresify fact cache receiver
* test OPTIONS endpoint
* Note: single fact view not implemented yet.
