azuredevops restapi does not return groups that are a member of a certain group

As a workaround, we can use the API to list all group and get group ID(Group ID in the response body is originId)

GET https://vssps.dev.azure.com/{Org name}/_apis/graph/groups?api-version=6.0-preview.1

Get the one group subjectDescriptor via below REST API

Note: subjectDescriptor is field value in the response body

GET https://vssps.dev.azure.com/{Org name}/_apis/graph/descriptors/{Group ID}

List group member

POST https://dev.azure.com/{Org name}/_apis/Contribution/HierarchyQuery?api-version=5.1-preview.1

Request Body

{
  "contributionIds": [
    "ms.vss-admin-web.org-admin-members-data-provider"
  ],
  "dataProviderContext": {
    "properties": {
      "subjectDescriptor": "{subjectDescriptor}",
      "sourcePage": {
        "url": "https://dev.azure.com/{Org name}/_settings/groups?subjectDescriptor={subjectDescriptor}",
        "routeId": "ms.vss-admin-web.collection-admin-hub-route",
        "routeValues": {
          "adminPivot": "groups",
          "controller": "ContributedPage",
          "action": "Execute"
        }
      }
    }
  }
}

Result:

Note: I recommend that you raise the initial issue to Developer Community, they will check it and contact to Product team.

Update1

Check group permission and get NamespaceId and Token.

GET https://dev.azure.com/{Org name}/_apis/Contribution/HierarchyQuery?api-version=6.0-preview

Request Body

{
  "contributionIds": [
    "ms.vss-admin-web.org-admin-groups-permissions-pivot-data-provider"
  ],
  "dataProviderContext": {
    "properties": {
      "subjectDescriptor": "{subjectDescriptor},
      "sourcePage": {
        "url": "https://dev.azure.com/v-viliu/_settings/groups?subjectDescriptor={subjectDescriptor}",
        "routeId": "ms.vss-admin-web.collection-admin-hub-route",
        "routeValues": {
          "adminPivot": "groups",
          "controller": "ContributedPage",
          "action": "Execute"
        }
      }
    }
  }
}

Get descriptor via below API

GET https://dev.azure.com/{Org name}/_apis/accesscontrollists/{securityNamespaceId}?api-version=6.0

Get group permission detail info

https://dev.azure.com/{Org name}/_apis/accesscontrollists/{securityNamespaceId}?token={Token}&descriptors={descriptor}&api-version=6.0

Note: This is binary code.

Update group permission.

POST https://dev.azure.com/{organization}/_apis/accesscontrollists/{securityNamespaceId}?api-version=6.0

We could get the request body through the API above, If you need to change permissions, you only need to change the value of field Allow and Deny.

You could also refer to this link.