Hi JustinWu,
Preferably, if you don't already, learn the language in the country of origin...at least enough to get around and learn the vocabulary for the industry in the native language. It will definitely go a long way because you will need to travel there if you are serious about sourcing directly.
I second what JohnP said. One of the hardest things I you will encounter sourcing directly from any farmer is travelling through the channels to export the coffee. Upon it arrival to the US it, usually, is not a problem. It takes a lot of time to get the supply chain to work efficiently, but when you start to deal with volume it is interesting because you are with it every step of the way. In addition, FDA Food Registration Facility Applications for the plant(s) in country of origin. If you can set those up your limitation is $, but then again so is everything haha.
Hope this helps a bit. Best of luck