how to register payment for an invoice in odoo using web apis

The action_register_payment method will check if active_ids is in context and if not it will return an empty string.
To pass the context you need to pass a dict just after the args list like they passed the fields argument to the read method in the official XML-RPC documentation:

models.execute_kw(db, uid, password, 'account.payment', 'action_register_payment', [[]], {'context': {"active_ids": [invoice_id]}}

The method returns an action as a dictionary which will lead to a:

TypeError: cannot marshal <class \'odoo.tools.misc.frozendict\'>

You can use the payment register wizard to create the invoice payments, you have to create a new payment register record then call the create_payments method.

Example:

payment_register_id = models.execute_kw(db, uid, password, 'account.payment.register', 'create', [{'journal_id': bank_journal_id, 'payment_method_id': payment_method_id, 'invoice_ids': [(4, invoice_id)]}])
models.execute_kw(db, uid, password, 'account.payment.register', 'create_payments', [[payment_register_id]])