In this paper, the nonsmooth compound transmission control protocol with the gentle random early detection system with the state-dependent round-trip delay is investigated in detail. Uniqueness of the positive equilibrium is proved firstly. Then, the closed approximate periodic solutions in this state-dependent delayed nonsmooth compound TCP with the GRED model are obtained by employing the harmonic balance and alternating frequency/time domain method. Then, we compare the results generated by numerical simulations with those of the closed approximate expressions obtained by HB-AFT. (...) It indicates that HB-AFT is simple, correct, and powerful for state-dependent delayed nonsmooth dynamical systems. Finally, we find the complicated dynamic: chaos. It is a grazing chaos with a hybrid property, i.e., where usually w oscillates at a very low frequency and q oscillates at a very high frequency. And, the route to chaos is a very rare route, i.e., the instantaneous and local transition of stable equilibrium to chaos. So, to the end of stability and good performance, we should adjust the parameters carefully to avoid the periodic and chaotic oscillations. (shrink)