how do i paint an outline when hovering over a qlistwidget item?

Cause

QPainter::drawRect draws a rectangle, that is slightly bigger (by exactly one pixel in height and width) than the painted area. The reason for this behavior could be seen in the way QPaintEngine draws a rectangle:

for (int i=0; i<rectCount; ++i) {
    QRectF rf = rects[i];
    QPointF pts[4] = { QPointF(rf.x(), rf.y()),
                       QPointF(rf.x() + rf.width(), rf.y()),
                       QPointF(rf.x() + rf.width(), rf.y() + rf.height()),
                       QPointF(rf.x(), rf.y() + rf.height()) };
    drawPolygon(pts, 4, ConvexMode);
}

QPaintEngine draws a closed polygon, starting at the point (x, y), going to (x + width, y), then to (x + width, y + height) and finally to (x, y + height). This looks intuitive enough, but let’s see what happens, if we substitute these variables with real numbers:

Say, we want to draw a 4x2 px rectangle at (0, 0). QPaintEngine would use the following coordinates: (0, 0), (4, 0), (4, 2) and (0, 2). Represented as pixels, the drawing would look like this:

So, instead of 4x2 px, we end up with a 5x3 px rectangle, i.e. indeed one pixel wider and taller.

You can further prove this by clipping the painter to option.rect before calling drawRect like this:

if (option.state & QStyle::State_MouseOver) {
    painter->setClipRect(option.rect);
    painter->drawRect(option.rect);
}

The result is clipped bottom and right edges of the outline (the very edges, we have expected to be within the painted area):

In any case, the part of the outline, that falls outside of the painted area, is not repainted correctly, hence the unwanted remains of the previous drawings in the form of lines.

Solution

Reduce the height and the width of the outline, using QRect::adjusted.

You might just write

painter->drawRect(option.rect.adjusted(0, 0, -1, -1));

However, this would work only for an outline, which is 1px thick and the devicePixelRatio is 1, as on a PC. If the border of the outline is thicker than 1px and/or the devicePixelRatio is 2, as on a Mac, of course more of the outline will stick out of the painted area, so you should take that into consideration and adjust the rectangle accordingly, e.g.:

int effectiveOutlineWidth = m_outineWidth*m_devicePixelRatio;
int tl = round(0.5*(effectiveOutlineWidth - 1));
int br = round(-0.5*effectiveOutlineWidth);

painter->drawRect(option.rect.adjusted(tl, tl, br, br));

m_outineWidth and m_devicePixelRatio are class members, representing the desired outline width, resp. the ratio between physical pixels and device-independent pixels for the paint device. Provided that you have created public setter methods for them, you could set their values like this:

auto *delegate = new MyDelegate(&w);

delegate->setOutlineWidth(1);
delegate->setDevicePixelRatio(w.devicePixelRatio());

w.setItemDelegate(delegate);

Example

Here is an example I have written for you to demonstrate how the proposed solution could be implemented:

#include <QApplication>
#include <QStyledItemDelegate>
#include <QListWidget>
#include <QPainter>

class MyDelegate : public QStyledItemDelegate
{
    int m_outineWidth;
    int m_devicePixelRatio;
public:
    
    MyDelegate(QObject *parent = nullptr) :
        QStyledItemDelegate(parent),
        m_outineWidth(1),
        m_devicePixelRatio(1) {
    }
    
    void paint(QPainter *painter, const QStyleOptionViewItem &option,
               const QModelIndex &index) const override {
        QStyledItemDelegate::paint(painter, option, index);
        
        if (option.state & QStyle::State_MouseOver) {
            int effectiveOutlineWidth = m_outineWidth*m_devicePixelRatio;
            int tl = round(0.5*(effectiveOutlineWidth - 1));
            int br = round(-0.5*effectiveOutlineWidth);
            
            painter->setPen(QPen(QBrush(Qt::red), m_outineWidth, Qt::SolidLine,
                                 Qt::SquareCap, Qt::MiterJoin));
            painter->drawRect(option.rect.adjusted(tl, tl, br, br));
        }
    }
    
    void setOutlineWidth(int outineWidth) {
        m_outineWidth = outineWidth;
    }
    
    void setDevicePixelRatio(int devicePixelRatio) {
        m_devicePixelRatio = devicePixelRatio;
    }
};

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    QListWidget w;
    auto *delegate = new MyDelegate(&w);
    
    delegate->setOutlineWidth(3);
    delegate->setDevicePixelRatio(w.devicePixelRatio());
    
    w.setItemDelegate(delegate);
    w.addItems(QStringList{"item1", "item2", "item3", "item4"});
    w.viewport()->setAttribute(Qt::WA_Hover);
    w.show();
    
    return a.exec();
}

Result

The provided example produces the following result for 3px thick outline on Windows: