We present the new constraints on the cosmic optical background (COB) obtained from an analysis of the Pioneer 10/11 Imaging Photopolarimeter (IPP) data. After careful examination of data quality, the usable measurements free from the zodiacal light are integrated into sky maps at the blue (~0.44 μm) and red (~0.64 μm) bands. Accurate starlight subtraction is achieved by referring to all-sky star catalogs and a Galactic stellar population synthesis model down to 32.0 mag. We find that the residual light is separated into two components: one component shows a clear correlation with thermal 100 μm brightness, while another betrays a constant level in the lowest 100 μm brightness region. Presence of the second component is significant after all the uncertainties and possible residual light in the Galaxy are taken into account, thus it most likely has the extragalactic origin (i.e., the COB). The derived COB brightness is (1.8 ± 0.9) × 10−9 and (1.2 ± 0.9) × 10−9 erg s−1 cm−2 sr−1 Å−1 at the blue and red band, respectively, or 7.9 ± 4.0 and 7.7 ± 5.8 nW m−2 sr−1. Based on a comparison with the integrated brightness of galaxies, we conclude that the bulk of the COB is comprised of normal galaxies which have already been resolved by the current deepest observations. There seems to be little room for contributions of other populations including “first stars” at these wavelengths. On the other hand, the first component of the IPP residual light represents the diffuse Galactic light (DGL)---scattered starlight by the interstellar dust. We derive the mean DGL-to-100 μm brightness ratios of 2.1 × 10−3 and 4.6 × 10−3 at the two bands, which are roughly consistent with the previous observations toward denser dust regions. Extended red emission in the diffuse interstellar medium is also confirmed.