Carrying abundant side information, knowledge graph (KG) has shown its great potential in enriching the sparsity of collaborative filtering (CF) for recommendation. Although graph neural networks (GNNs) have been successfully employed to learn user preferences from KG and CF signals simultaneously, most models suffer from inferior performance due to their deficient designs, i.e., 1) formulating no distinction between users, items and KG entities, 2) confounding KG signals with CF signals and 3) completely neglecting the effects of edges, which is vital for graph information propagation. In this paper, we propose a quad-channel graph model (X-2ch) to tackle these problems. First, rather than lodging KG entities on graph as nodes, X-2ch distills KG information and embeds them as edge attributes in a bi-directional manner to model the natural user-item interaction process. Second, X-2ch introduces a novel quad-channel learning scheme, including a collaborative user-item update and a CF-KG attentive propagation, to holistically capture the interconnectivity of users and items while preserving their distinct properties. Experiments on two real-world benchmarks show substantial improvement over the state-of-the-art baselines.