is there any kql query to limit the number of sub results i get per category?

Here you go:

let NumItemsPerCategory = 3;
datatable(ID:long, Category:string, Value:guid)
[
    1, "A", guid(40b73f8f-78d2-4eae-bd5b-b3e00f38ac33), 
    2, "A", guid(043ee507-aadf-4453-bcc6-d8f4f541b043), 
    3, "A", guid(f71d3cc0-ce46-474f-9dcd-f3883fa08859), 
    4, "A", guid(bf259fc8-e9fe-4a99-a296-ca81e1fa250a), 
    5, "B", guid(d8ee3ac7-da76-4e87-a9ed-e5a37c943ad2), 
    6, "B", guid(282e74ff-3b71-407c-a2a7-92bb1cb17b27), 
]
| summarize PackedItems = make_list(pack_all(), NumItemsPerCategory) by Category
| project-away Category
| mv-expand PackedItem = PackedItems
| evaluate bag_unpack(PackedItem)
| project-away PackedItems

Result:

| ID | Category | Value                                |
|----|----------|--------------------------------------|
| 1  | A        | 40b73f8f-78d2-4eae-bd5b-b3e00f38ac33 |
| 2  | A        | 043ee507-aadf-4453-bcc6-d8f4f541b043 |
| 3  | A        | f71d3cc0-ce46-474f-9dcd-f3883fa08859 |
| 5  | B        | d8ee3ac7-da76-4e87-a9ed-e5a37c943ad2 |
| 6  | B        | 282e74ff-3b71-407c-a2a7-92bb1cb17b27 |