static navbar shakes horizontaly on leaving modal

Inspecting your codepen, I observed that the margin-left and margin-right of the child element of the #header increases, when the modal opens.

When you leave modal, margin-left and margin-right are returning back to their initial values, causing the shake.

The fixed position seems to be the root of cause. Thus, I would go with position: sticky instead and add some margin top in the .inner-pages.

For example,

.fixed-top {
    position: sticky;
    top: 0;
    right: 0;
    left: 0;
    z-index: 1030;
}

.inner-page {
    margin-top: 50px;
}