Can Two Python Argparse Objects Be Combined?

December 27, 2023 Post a Comment

I have an object A which contains parserA - an argparse.ArgumentParser object There is also object B which contains parserB - another argparse.ArgumentParser Object A contains an i

Solution 1:

argparse was developed around objects. Other than a few constants and utility functions it is all class definitions. The documentation focuses on use rather than that class structure. But it may help to understand a bit of that.

parser = argparse.ArgumentParser(...)

creates a parser object.

arg1 = parser.add_argument(...)

creates an argparse.Action (subclass actually) object and adds it to several parser attributes (lists). Normally we ignore the fact that the method returns this Action object, but occasionally I find it helpful. And when I build a parser in an interactive shell I see a this action.

args = parser.parse_args()

runs another method, and returns an namespace object (class argparse.Namespace).

The group methods and subparsers methods also create and return objects (groups, actions and/or parsers).

The ArgumentParser method takes a parents parameter, where the value is a list of parser objects.

With

parsera = argparse.ArgumentParser(parents=[parserb])

during the creation of parsera, the actions and groups in parserb are copied to parsera. That way, parsera will recognize all the arguments that parserb does. I encourage you to test it.

But there are a few qualifications. The copy is by reference. That is, parsera gets a pointer to each Action defined in parserb. Occasionally that creates problems (I won't get into that now). And one or the other has to have add_help=False. Normally a help action is added to a parser at creation. But if parserb also has a help there will be conflict (a duplication) that has to be resolved.

But parents can't be used if parsera has been created independently of parserb. There's no existing mechanism for adding Actions from parserb. It might possible to make a new parser, with both as parents

parserc = argparse.ArgumentParser(parents=[parsera, parserb])

I could probably write a function that would add arguments from parserb to parsera, borrowing ideas from the method that implements parents. But I'd have to know how conflicts are to be resolved.

Look at the argparse._ActionsContainer._add_container_actions to see how arguments (Actions) are copies from a parent to a parser. Something that may be confusing is that each Action is part of a group (user defined or one of the 2 default groups (seen in the help)) in addition to being in a parser.

Another possibility is to use

[argsA, extrasA] = parserA.parse_known_args()
[argsB, extrasB] = parserB.parse_known_args()  # uses the same sys.argv # or
args = parserB.parse_args(extrasA, namespace=argsA)

With this each parser handles the arguments it knows about, and returns the rest in the extras list.

Unless the parsers are designed for this kind of integration, there will be rough edges with this kind of integration. It may be easier to deal with those conficts with Arnial's approach, which is to put the shared argument definitions in your own methods. Others like to put the argument parameters in some sort of database (list, dictionary, etc), and build the parser from that. You can wrap parser creation in as many layers of boilerplate as you find convenient.

Solution 2:

You can't use one ArgumentParser inside another. But there is a way around. You need to extract to method code that add arguments to parser. Then you will be able to use them to merge arguments in parser. Also it will be easer to group arguments (related to their parsers). But you must be shore that sets of arguments names do not intersect.

Example:

foo.py:

def add_foo_params(group):
   group.add_argument('--foo', help='foo help')

if __name__ = "__main__":  
   parser = argparse.ArgumentParser(prog='Foo')

boo.py

def add_boo_params(group):
   group.add_argument('--boo', help='boo help')

if __name__ = "__main__":  
   parser = argparse.ArgumentParser(prog='Boo')

fooboo.py

from foo import add_foo_params
   from boo import add_boo_params

   if __name__ = "__main__":  
       parser = argparse.ArgumentParser(prog='FooBoo')
       foo_group = parser.add_argument_group(title="foo params")
       boo_group = parser.add_argument_group(title="boo params")

       add_foo_params( foo_group )
       add_boo_params( boo_group )

Solution 3:

For your use case, if you can, you could try simply sharing the same argparse object between classes via a dedicated method. Below is based on what it seems like your situation is.

import argparse

classB(object):

  def__init__(self, parserB=argparse.ArgumentParser()):
    super(B, self).__init__()
    self.parserB = parserB

  defaddArguments(self):
    self.parserB.add_argument("-tb", "--test-b", help="Test B", type=str, metavar="")
    #Add more arguments specific to BdefparseArgs(self):
    return self.parserB.parse_args()

classA(object):

  def__init__(self, parserA=argparse.ArgumentParser(), b=B()):
    super(A, self).__init__()
    self.parserA = parserA
    self.b = b

  defaddArguments(self):
    self.parserA.add_argument("-ta", "--test-a", help="Test A", type=str, metavar="")
    #Add more arguments specific to AdefparseArgs(self):
    return self.parserA.parse_args()

  defmergeArgs(self):
    self.b.parserB = self.parserA
    self.b.addArguments()
    self.addArguments()

Code Explanation:

As stated, in the question, object A and object B contain their own parser objects. Object A also contains an instance of object B.
The code simply separates the anticipated flow into separate methods so that it is possible to keep adding arguments to a single parser before attempting to parse it.

Test Individual

a = A()
a.addArguments()
print(vars(a.parseArgs()))

# CLI Command
python test.py -ta "Testing A"# CLI Result
{'test_a': 'Testing A'}

Combined Test

aCombined = A()
aCombined.mergeArgs()
print(vars(aCombined.parseArgs()))

# CLI Command
testing -ta "Testing A" -tb "Testing B"# CLI Result
{'test_b': 'Testing B', 'test_a': 'Testing A'}

Additional

You can also make a general method that takes variable args, and would iterate over and keep adding the args of various classes. I created class C and D for sample below with a general "parser" attribute name.

Multi Test

# Add method to Class AdefmergeMultiArgs(self, *objects):
    parser = self.parserA
    forobjectin objects:
        object.parser = parser
        object.addArguments()
    self.addArguments()

aCombined = A()
aCombined.mergeMultiArgs(C(), D())
print(vars(aCombined.parseArgs()))

# CLI Command
testing -ta "Testing A" -tc "Testing C" -td "Testing D"# CLI Result
{'test_d': 'Testing D', 'test_c': 'Testing C', 'test_a': 'Testing A'}

Solution 4:

Yes they can be combined, do this:

Here is a function that merges two args:

defmerge_args_safe(args1: Namespace, args2: Namespace) -> Namespace:"""
    Merges two namespaces but throws an error if there are keys that collide.

    ref: https://stackoverflow.com/questions/56136549/how-can-i-merge-two-argparse-namespaces-in-python-2-x
    :param args1:
    :param args2:
    :return:
    """# - the merged args# The vars() function returns the __dict__ attribute to values of the given object e.g {field:value}.
    args = Namespace(**vars(args1), **vars(args2))
    return args

test

def merge_args_test():
    args1 = Namespace(foo="foo", collided_key='from_args1')
    args2 = Namespace(bar="bar", collided_key='from_args2')

    args = merge_args(args1, args2)
    print('-- merged args')
    print(f'{args=}')

output:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1202, in <module>
    merge_args_test()
  File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1192, in merge_args_test
    args = merge_args(args1, args2)
  File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1116, in merge_args
    args = Namespace(**vars(args1), **vars(args2))
TypeError: argparse.Namespace() got multiple values for keyword argument 'collided_key'
python-BaseException

you can find it in this library: https://github.com/brando90/ultimate-utils

If you want to have collisions resolved do this:

defmerge_two_dicts(starting_dict: dict, updater_dict: dict) -> dict:"""
    Starts from base starting dict and then adds the remaining key values from updater replacing the values from
    the first starting/base dict with the second updater dict.

    For later: how does d = {**d1, **d2} replace collision?

    :param starting_dict:
    :param updater_dict:
    :return:
    """new_dict: dict = starting_dict.copy()   # start with keys and values of starting_dict
    new_dict.update(updater_dict)    # modifies starting_dict with keys and values of updater_dictreturn new_dict

defmerge_args(args1: Namespace, args2: Namespace) -> Namespace:"""

    ref: https://stackoverflow.com/questions/56136549/how-can-i-merge-two-argparse-namespaces-in-python-2-x
    :param args1:
    :param args2:
    :return:
    """# - the merged args# The vars() function returns the __dict__ attribute to values of the given object e.g {field:value}.merged_key_values_for_namespace: dict = merge_two_dicts(vars(args1), vars(args2))
    args = Namespace(**merged_key_values_for_namespace)
    return args

test:

def merge_args_test():
    args1 = Namespace(foo="foo", collided_key='from_args1')
    args2 = Namespace(bar="bar", collided_key='from_args2')

    args = merge_args(args1, args2)
    print('-- merged args')
    print(f'{args=}')
    assert args.collided_key == 'from_args2', 'Error in merge dict, expected the second argument to be the one used' \
                                                 'to resolve collision'

Python Dummy