Christian Herdtweck [Thu, 25 Apr 2024 10:40:41 +0000]
Add tool to query state of system service
I have written this or very similar code a few times, time to centralize
it.
Christian Herdtweck [Wed, 6 Mar 2024 10:35:38 +0000]
Raise error if email transfer wait reaches timeout
Christian Herdtweck [Fri, 24 Nov 2023 14:46:46 +0000]
Prepare adaptation of unittests for python 3.11
Apparently, error tracebacks will have extra lines with pointers to the
position in the line where the error occurred.
Christian Herdtweck [Fri, 24 Nov 2023 14:11:57 +0000]
Create 2 unit tests for wait_for_generate
Christian Herdtweck [Fri, 24 Nov 2023 14:11:43 +0000]
Improve reliability of wait_for_generate
We encountered race conditions in QA where generate had finished, but while
we checked for generate_offline a new generate was scheduled. Do both at
the same time, now.
Christian Herdtweck [Wed, 6 Sep 2023 07:32:23 +0000]
Minor improvement of cnfvar diff output
Add unittest for new output
Christian Herdtweck [Thu, 13 Jul 2023 10:24:30 +0000]
Reduce logging noise produced by arnied_wrapper
At least the log messages that appear repeatedly should be reduced to
"debug" level.
Christian Herdtweck [Fri, 30 Jun 2023 14:36:36 +0000]
Increase version to 1.7.2
Christian Herdtweck [Tue, 4 Jul 2023 14:03:10 +0000]
Make CnfVar lists comparable
Add a mixin with a compare() function to BaseCnfList that returns an
instance of CnfDiff to simplify printing diffs.
Add unittests for new functionality.
Christian Herdtweck [Mon, 19 Jun 2023 09:40:50 +0000]
Move mail util for waiting for quarantine from QA
Christian Herdtweck [Tue, 8 Nov 2022 10:37:09 +0000]
Add 3 unittests with fixed log file contents
Add some stability/reliability/reproducability to log parsing unit tests.
Also: fix type documentation in LogParser doc string
Thomas Jarosch [Tue, 8 Nov 2022 08:58:25 +0000]
Add new VPNCONN_PROTO default VPNCONN cnfvar
Change needed due to upcoming Wireguard support.
Thomas Jarosch [Tue, 8 Nov 2022 08:57:02 +0000]
Adjust SYS_LOG_PATTERN to parse single digit day number
Noticed by this failing unit test:
FAIL: test_parse_messages (test_log_read.LogReadTester)
Try parsing first 100 lines of messages if running on linux
----------------------------------------------------------------------
Traceback (most recent call last):
File "pyi2ncommon/test/test_log_read.py", line 268, in test_parse_messages
self.assertIsNotNone(data, f"Failed to parse {parser.last_unparsed_line}")
AssertionError: unexpectedly None : Failed to parse Nov 7 09:24:31 storm kernel: Linux version 6.0.5-200.fc36.x86_64 (mockbuild@bkernel02.iad2.fedoraproject.org) (gcc (GCC) 12.2.1
20220819 (Red Hat 12.2.1-2), GNU ld version 2.37-36.fc36) #1 SMP PREEMPT_DYNAMIC Wed Oct 26 15:55:21 UTC 2022
Christian Herdtweck [Wed, 18 May 2022 11:53:50 +0000]
Increase version to 1.7.1
Christian Herdtweck [Thu, 19 May 2022 11:30:25 +0000]
Merge branch 'api-doc-improvements'
Christian Herdtweck [Wed, 18 May 2022 09:15:34 +0000]
Remove api doc headers
These headers (usually "SUMMARY", "CONTENTS", "INTERFACE") were inherited
from QA in a few places but not used consistently and in most places much
too prominent considering the little text they structured. Removed them in
most places and in some replaced them with other structuring elements like
lists.
Christian Herdtweck [Wed, 18 May 2022 08:36:50 +0000]
Improve table of contents
Christian Herdtweck [Tue, 17 May 2022 12:43:11 +0000]
Fix errors in api doc creation
There were a number of warnings and errors when creating docs, resulting in
html docs that were screwed up in a few places. Fixed those.
Christian Herdtweck [Thu, 19 May 2022 11:23:42 +0000]
Merge branch 'log-parser'
Christian Herdtweck [Thu, 19 May 2022 11:16:50 +0000]
Add patterns for common linux log lines
Needed to analyze large proxy logs in support. This helped.
Syslog pattern simplifies testing of log_read.
Christian Herdtweck [Thu, 19 May 2022 10:19:17 +0000]
Create more convenient constructor for LogParser
Christian Herdtweck [Wed, 18 May 2022 11:48:50 +0000]
Return matches or None from LogParser iteration
Yielding match-object or string is bad style. Iteration over LogParser now
yields either match-object or None. If None is returned (i.e. parsing line
failed), then the problematic input is saved in LogParser object's attribute
last_unparsed_line (until next matching fails).
To make this work we need to run prepare_result on every single line, so
prepare_result cannot return multiple lines at once. Therefore, made
prepare_result also a generator (which makes more sense, anyway).
Christian Herdtweck [Thu, 19 May 2022 11:16:59 +0000]
Implement simple reading of stationary logs
Change default behaviour: assume log is not being written to any more, so
iteration over log lines ends once end is reached. Still allow for original
"following" behaviour with changed parameter.
Christian Herdtweck [Mon, 25 Apr 2022 12:51:19 +0000]
Fix error in log reader
When receiving several lines, the LogParser would only parse and
return the first. Fixed that.
Christian Herdtweck [Thu, 19 May 2022 09:13:44 +0000]
Merge branch 'cnfvar-deprecations'
Plamen Dimitrov [Thu, 21 Apr 2022 07:05:23 +0000]
Deprecate any arnied wrapper cnfvar functionality
This also includes dropping a few functions using hard-coded
{get|set|unset}_cnf binary calls for functionality that is
simple enough to be taken care of by client code.
It leaves us only with remainder functions that we would like
to eventually get either through the arnied API or through some
other module specializing on top of the arnied API.
Plamen Dimitrov [Mon, 18 Apr 2022 01:21:27 +0000]
Reorder and complete the string and file serialization methods
We were missing to_cnf_string() methods which are usually replaced
by the simple `str(Cnf(...))` but it is still better to add them
for completeness in code relying on the full mix of serialization
methods at least for the sake of syntactic consistency and thus
better readability in some particular use cases.
Plamen Dimitrov [Sat, 16 Apr 2022 15:18:22 +0000]
Deprecate "to" and privatize "from" cnf structure methods
These are no longer officially used by any client code and are now
restricted only to one directional internal use when it comes to
the old functionality of reading cnfvars from strings or JSON data
which were originally developed to use the same format (cnf structure).
Plamen Dimitrov [Thu, 28 Apr 2022 06:13:08 +0000]
Drop localized dialout cnfvar complexity for the easier cnf store
This is the final step towards a fully adapted dialout module and
gets rid simultaneously of outdated arnied cnfvar calls as well as
outdated cnf structure parsing in the new cnfvar API, both of which
will follow next.
Plamen Dimitrov [Wed, 27 Apr 2022 04:01:16 +0000]
Convert the old cnfvar module into a cnfvar string parsing module
In other words, we drop any redundant and duplicate functionality
and leave out only the string parsing methods which are still used
by the cnfvar subpackage and keep their corresponding unit tests.
Plamen Dimitrov [Sat, 9 Apr 2022 05:06:21 +0000]
Drop previous simple cnf module and class throughout the codebase
Within pyi2ncommon this is only a matter of isolated narrowly used
code within the dial module.
Plamen Dimitrov [Thu, 12 May 2022 04:21:23 +0000]
Merge branch 'cnfline-templates' into cnfvar-deprecations
This subbranch merges the mk_config templates and the cnfline classes
into single signature functions with maximum use case versatility.
Plamen Dimitrov [Tue, 5 Apr 2022 03:29:01 +0000]
Replace the mk_config with a sample minimal cnfvar templates module
The following commit provides an example for turning the partially
developed (and always incomplete for new use cases) mk_config functions
into ones that only provide a minimal set of defaults and can still
be used as one-liners by client/user code. The example implementation
at the moment is for a simple and a user template cnfvars and will
later on be extended to groups and all other major cnfvars.
Plamen Dimitrov [Tue, 17 May 2022 03:03:10 +0000]
Add a minimal coverage unit test for runtime breakage detection
Since the template fiels are mostly data loaded from templates, the
idea is to at least go through all of them and check for correct
class of the generated cnfvars and for defaults dictionaries making
sure the templates module is not populated with anything else.
Plamen Dimitrov [Tue, 5 Apr 2022 01:44:57 +0000]
Drop any cnfvar manipulation from the generic mail utils
The conclusion from previous attempts to convert any user cnfvar
manipulation to dynamic mode was that each user could be specific
to the use case where it is created so handling this in the mail
utils should be replaced by direct calls to the new cnfvar API.
Plamen Dimitrov [Thu, 12 May 2022 04:00:58 +0000]
Add minimal defaults modification based on cnfvar type/mode
Some cnfvars like providers have different modes and some cnfvars
like nics have differnet types that change a large number of default
child variables. This in turn results in large modification overhead
when changing their type/mode, the largest of which could be handled
by the templates themselves.
Christian Herdtweck [Mon, 2 Dec 2019 12:08:24 +0000]
Change creation of mail users to dynamic
This is more easily extendable and avoids old bad habit of passing down
huge param dicts to very specific functions like this one.
User creation is hard to test, so there is no unittest for this yet. Manual
tests succeeded. Have to adapt QA tests and test there further.
Additionally to the original conversion, default mail users don't have
to be administrators.
Plamen Dimitrov [Thu, 12 May 2022 03:57:33 +0000]
Unify the template arguments across various template cnfvars
There are no obligatory fields as whether any of them are used or
required depends entirly on the cnfvar type/mode for each template.
For instance, AD users may not have password field, GWINLAN providers
may not have local IP, DSL and PPTP providers may not have provider IP,
and protocol forwarding might not have source and destination ports.
It is thus better to unify the function signatures entirely and later
provide some extra facilitation for different cnfvar types/modes.
Plamen Dimitrov [Thu, 12 May 2022 05:06:35 +0000]
Sanitize all collected default values to empty fields
All values are better off being empty rather than filled with noisy
sample data that could be wrongfully introduced on the user side.
Plamen Dimitrov [Thu, 12 May 2022 05:03:11 +0000]
Drop any None and thus previously required default values
There can be no "None" cnfvar values as these will be automatically
converted to "None" strings. Empty strings are thus their correct
equivalent after the cnfline classes migration.
As no child cnfvars could be considered obligatory and whether they
are required depends entirely on choices for other child cnfvars,
we will not try to implement any option for an early fail and simply
delegate any further validation to the production code.
Plamen Dimitrov [Thu, 12 May 2022 05:01:01 +0000]
Use better and fresh/recent UI defaults for all templates
In particular, German locale for user cnfvars is a better choice
considering our current customer base and some values were not
actual or at least recent UI defaults before the migration.
Plamen Dimitrov [Sat, 9 Apr 2022 05:05:30 +0000]
Drop the cnfline subpackage and reuse all of its defaults for templates
In particular, add defaults for already available templates through
the old mk_config but also migrate the cnfline ones in the same
location.
Plamen Dimitrov [Sat, 9 Apr 2022 05:06:08 +0000]
Use group (minimal GUI) defaults similarly to the user template
This provides a complete user-group combination for testing of
the template concept before migrating more templates code.
Note that the UI-defaults for group child cnfvars are not the same
as the ones provided in the build_group module which seem to be
too few and the minimal needed for the configuration to be accepted.
When creating an empty GROUP in the UI, these fields are populated:
1 GROUP,104: "ganz neu"
2 (1) GROUP_ACCESS_GO_ONLINE_ALLOWED,0: "1"
3 (1) GROUP_COMMENT,0: ""
4 (1) GROUP_EMAILFILTER_BAN_FILTERLIST_REF,0: "-1"
5 (1) GROUP_EMAIL_RELAY_RIGHTS,0: "RELAY_FROM_INTRANET"
6 (1) GROUP_PROXY_PROFILE_REF,0: "1"
So there is a tad more than just "GROUP_COMMENT" and following the
UI these additional fields should be added and similarly populated.
Plamen Dimitrov [Thu, 7 Apr 2022 04:44:29 +0000]
Introduce use of defaults i.e. UI-default rather than minimal user cnfvars
Christian Herdtweck [Thu, 19 May 2022 08:53:16 +0000]
Merge branch 'cnfvar-review-improvements' into master
Plamen Dimitrov [Thu, 21 Apr 2022 07:48:40 +0000]
Wait for arnied in addition to generate to preserve backward compatibility
We simplify this a bit though by waiting for the arnied availability
just once during initialization of the cnf store and not on each
get/set/unset_cnf call (now query, commit, and delete calls).
As this is done for backward compability which is guaranteed for
the binary cnfvar driver only, we leave a TODO where we will have
to consider what is the best future for the varlink driver: either
busy-loop on socket availability to once again wait with a timeout
or drop this assumption entirely for better control and easier use.
Plamen Dimitrov [Wed, 20 Apr 2022 07:45:23 +0000]
Restore support for zero instance filtering in binary cnf store queries
Some cnfvars like NICs support zero instance numbers and without
this fix the query would simply return all cnfvars in a cnf list
if we need `get_cnf CNFVAR 0` from the query.
Plamen Dimitrov [Wed, 27 Apr 2022 03:46:35 +0000]
Fix and improve readability of cnfvar API backend type logging
The original logging meant to use instances and not entire classes
resulting in unreadable "type" log messages.
Plamen Dimitrov [Wed, 4 May 2022 08:10:24 +0000]
Add case-insensitive replace() behavior to str-inheriting CnfName
Plamen Dimitrov [Sat, 16 Apr 2022 15:11:32 +0000]
Implement add() to add cnfvars to cnfvar lists directly from tuples
The CnfList class already has some functionality for initialization
directly from tuples and the Cnf class already has some functionality
like add_child() for adding a cnfvar from a tuple to Cnf instance's
CnfList instance children. To complete the syntactic sugar circle,
also provide the further possibility to add a CnfList item straight
from a tuple using add() instead of the more raw and manual append()
which can also handle both Cnf instances and tuples similarly to the
previous two options above.
Plamen Dimitrov [Sat, 16 Apr 2022 15:12:05 +0000]
Fix inconsistent cnf structure method naming
Part of the API methods were using "from/to_cnfvar_dict" while the
other part were using "from/to_cnf_structure".
Plamen Dimitrov [Thu, 21 Apr 2022 07:05:35 +0000]
Add a shortcut to obtain the instance of a possibly existing cnfvar
This is added in the cnf list query mixin which is where we have
all shortcut, syntactic sugar, and ease of use methods.
Plamen Dimitrov [Mon, 25 Apr 2022 09:35:28 +0000]
Revert the default instance number back to zero
These types of cnfvars are much more ubiquitous in the code base
and cnfvars that could be instantiated in ranges of instance numbers
are typically instantiated via templates (cnfvar.templates) due to
their size complexity thus not really benefiting from a default -1
instance here.
Christian Herdtweck [Wed, 6 Apr 2022 10:53:10 +0000]
Improve creation of docs
Want to have the name "pyi2ncommon" in there, not have it replaced by "src".
Realized that modules.rst is auto-created, so does not belong into git.
Christian Herdtweck [Wed, 6 Apr 2022 10:44:17 +0000]
Update deprecation warnings
Parts of arnied_wrapper will have to stay. However, simple_cnf,
mk_config and cnfline are to be removed, too.
Also make other warnings a proper reST note.
Christian Herdtweck [Wed, 6 Apr 2022 10:43:03 +0000]
Update purpose in module descriptions
After a long discussion between Tom, Plamen, and Christian about location
and form of config-related functionality, it was realized that pyi2ncommon
is not just a collection of helpers for internal use, but also an interface
for "external" users of Intra2net system code like QA. Adjust description
in spec file and doc main page accordingly.
Plamen Dimitrov [Tue, 12 Apr 2022 07:32:43 +0000]
Update mail utils folder name assumptions from Objekte to Elemente
"Gesendete Objekte" was the old name from Outlook 2003-2007 and was
changed to "Gesendete Elemente" in Outlook 2010 onwards. We aligned
ourselves with Outlook since it's the most used email client with
the IBS.
Christian Herdtweck [Fri, 1 Apr 2022 11:29:19 +0000]
Create argparse type "existing_dir[_empty]"
Programs often have a file or dir as argument, check in argparse right
away whether that actually exists. Optionally, accept an empty value to
use a default value.
Christian Herdtweck [Mon, 4 Apr 2022 15:45:14 +0000]
Clean up, remove compat with py < 3.6
Remove __future__ statements and other remainders of python2-compatibility.
Remove checks for python version <3.6, assume we are running on python 3.6
or above.
Also run a major pylint on all the code and fix lots of things, including
many typos, missing paramters, strange whitespace, "r" in front of regex
string literals, ...
However, ignore deprecated modules.
Christian Herdtweck [Mon, 4 Apr 2022 12:50:49 +0000]
Increase version to 1.7.0
Christian Herdtweck [Fri, 1 Apr 2022 14:47:59 +0000]
Little fixes found during cnfvar-api-Review
Christian Herdtweck [Tue, 5 Apr 2022 07:20:23 +0000]
Merge branch 'cnfvar-api'
Christian Herdtweck [Fri, 1 Apr 2022 14:41:42 +0000]
Review: a few fixes
functional changes:
* get to run on python < 3.8
* add missing args for shadowed functions startswith/endswith
* some more
non-functional changes:
* add some doc strings (mostly arnied_api)
* wrap very long lines
* add some whitespace to make linter happier
* fix a few typos
* remove unnecessary "object" super class
* fix api documentation
* add diagram that Samir had created to api doc
Christian Herdtweck [Fri, 1 Apr 2022 13:31:41 +0000]
Deprecate cnfvar_old and arnied_wrapper
Just via text in doc string, do not raise Warnings yet
Samir Aguiar [Sun, 15 Aug 2021 17:43:00 +0000]
Add a new cnfvar2 API
It is composed of three main modules:
- binary: wrappers around the get_ and set_cnf binaries
- model: provides Cnf and CnfList classes, which are Python
representation of cnfvars with dozens of facility methods to
manipulate them
- store: classes implemented using the repository pattern that
run queries against the arnied API or the binaries using the above
module and return CnfList instances
Samir Aguiar [Mon, 14 Feb 2022 18:51:27 +0000]
Add a Python version of arnied varlink interface
Also add as many typings as possible.
Can be easily used from within an IBS VM that has the varlink module
installed:
```
from pyi2ncommon.arnied_api import Arnied, GetCnfQuery
cnfs = Arnied.get_cnf(GetCnfQuery("USER"))
user = cnfs.vars[0]
assert user.name == "USER"
assert user.instance == 1
assert user.data == "admin"
```
Samir Aguiar [Thu, 17 Mar 2022 21:14:05 +0000]
Rename cnfvar module to cnfvar_old
The new API will replace it and we want to prevent users from being
confused about which to use.
Samir Aguiar [Mon, 23 Aug 2021 03:26:48 +0000]
Various fixes to the cnfvar module
- fix infinite loop with blank lines
- do not drop parent and comment values when normalizing
- fix error when first child begins with whitespaces
- format, restructure and document the module
Christian Herdtweck [Fri, 1 Apr 2022 11:30:44 +0000]
Ignore JetBrains IDE dir
Christian Herdtweck [Wed, 10 Nov 2021 09:31:53 +0000]
Version 1.6.7
Christian Herdtweck [Wed, 10 Nov 2021 09:29:57 +0000]
Merge branch 'logger-accept-handlers'
Christian Herdtweck [Thu, 19 Dec 2019 16:16:35 +0000]
Fix comments and spacing
Christian Herdtweck [Thu, 19 Dec 2019 16:16:20 +0000]
Add methods to more easily add handlers
Christian Herdtweck [Thu, 19 Dec 2019 16:12:20 +0000]
Create unittest for log_helper from test functions
Christian Herdtweck [Fri, 9 Aug 2019 13:19:15 +0000]
Fix a small bug in log helper
Christian Herdtweck [Fri, 9 Aug 2019 13:19:00 +0000]
Add a NullHandler if no handler is supplied to i2n logger
Christian Herdtweck [Fri, 9 Aug 2019 10:57:06 +0000]
Add option to add specified handlers to logger
Christian Herdtweck [Mon, 8 Nov 2021 15:12:46 +0000]
Merge branch 'log-read-improve'
Christian Herdtweck [Wed, 12 Dec 2018 10:02:07 +0000]
log_read: correct error message
Christian Herdtweck [Wed, 12 Dec 2018 09:54:56 +0000]
log_read: return/yield all of input args from iteration
Christian Herdtweck [Wed, 12 Dec 2018 09:49:48 +0000]
log_read: Make names of class variables clearer
Christian Herdtweck [Tue, 22 Jan 2019 09:50:41 +0000]
log_read: Fix import order and return from prepare_result
Christian Herdtweck [Tue, 22 Jan 2019 09:50:10 +0000]
log_read: create/Fix docstrings
Including fixes inspired by Plamen's review.
Christian Herdtweck [Mon, 8 Nov 2021 15:01:25 +0000]
log_read: One more check for test
Christian Herdtweck [Mon, 8 Nov 2021 14:53:58 +0000]
log_read: Deal with rare case of wrong fstat at test start
Christian Herdtweck [Mon, 8 Nov 2021 13:41:18 +0000]
log_read: Just warn when file shrinks, try reading anyway
Christian Herdtweck [Mon, 8 Nov 2021 08:26:52 +0000]
Fix get_cnf if working on local machine
get_cnf always returns latin1-encoded output, but wrapper decoded shell-output
with system-default-encoding (usually utf8). Fix that
Christian Herdtweck [Mon, 8 Nov 2021 08:24:40 +0000]
Add argparse-checker for file allowing empty
Often one needs to give as argument an existing file, so we have a check for
that, that can be used with argparse. However, sometimes you want to give
users the option to not give a filename at all, which is what this new
checker allows.
Christian Herdtweck [Mon, 8 Nov 2021 08:23:01 +0000]
Fix some whitespace and line breaks
Christian Herdtweck [Wed, 15 Sep 2021 08:07:10 +0000]
Reduce assumptions on stdout
We might be running in an environment that captures stdout
by using some replacement class. Do not fail in that case
but just disble coloring
Christian Herdtweck [Wed, 15 Sep 2021 08:05:54 +0000]
Do not assume running tests in English system
Error message might differ greatly. This will have to be
adjusted further, works now for German and English
Christian Herdtweck [Wed, 15 Sep 2021 08:04:25 +0000]
Do not re-run tests if they failed already
Christian Herdtweck [Fri, 10 Sep 2021 12:02:08 +0000]
Bump release, now version 1.6.6
Christian Herdtweck [Fri, 10 Sep 2021 11:58:40 +0000]
Merge branch 'arnied-wait-socket'
Samir Aguiar [Thu, 12 Aug 2021 21:26:25 +0000]
Use the already defined function to wait for generate
Samir Aguiar [Tue, 31 Aug 2021 22:39:04 +0000]
Improve documentation of arnied wrapper and tests
Samir Aguiar [Thu, 12 Aug 2021 12:54:14 +0000]
Add a new function to wait for arnied to be ready
Usually code depending on arnied checks whether its process is running
before proceeding, which can lead to a race condition when it is not
yet fully started. The new function makes use of a novel CLI option
in the arnied helper that blocks execution until arnied is indeed
ready to be used.
Samir Aguiar [Wed, 1 Sep 2021 01:23:04 +0000]
Fix unit tests
The dummy provider CNF variable was not in the correct form.
Also drop the TODO regarding raising an error when failing to go
online. The `run_cmd` function, which is mocked in the unit tests,
does check for the exit code of commands before returning.
Christian Herdtweck [Wed, 8 Apr 2020 15:54:51 +0000]
Add license and doc to rpm package
Failed to do this using setup.py and hacks in make_dist.py,
at least when doc and license files should not end up in
site-packages with the source code. Therefore change (and
simplify) the whole build process by re-creating
make_dist.py to just create a tarball that is compatible
with standard python spec files (like that of our QA) and
provide such a simple spec file to be used with rpmbuild.
Christian Herdtweck [Thu, 2 Sep 2021 08:03:07 +0000]
Remove superfluous doc/README