Don't think about what the other people are doing because they may be selling crap coffee or coffee that costs a lot more than you. So consider your own situation. Figure out how much money you want/need to make. You can't work for free.
After that it is simple math -- your price + distributor margin = wholesale price even if you're doing the distribution right now, because chances are you won't be doing that in the future.
Now look at your competitor prices. If you've got room to go up, do so. Don't go down, remember you set the price you need to make. Don't undercut yourself. If the wholesaler wants a better price move on.
Also the wholesaler will press you for $1 off specials, rebates, etc. It all comes out of your pocket. Remember that you're an artisan roaster, not a factory roaster, so don't go down that road. When they press you on it just say that really, you need to raise your prices based on your coffee quality, and are doing the best you can to hold prices the way they are.
One caveat here: think ahead. For instance what about packaging? When we serviced stores ourselves we delivered per bag. Then we got accounts that wanted us to ship them boxes. So one example is that we have a box made that holds 8 coffee bags. Then that box goes into a larger box that holds 12 of the 8-bag boxes (96 bags of coffee). This is sized this way because the courier service here has a max weight of 30kg (another thing to think about). So all those boxes cost a fair amount. Divide that by 96 bags of coffee and add that to your cost too. Not saying that is the packaging configuration you'll have but if you get to any size you'll be shipping coffee so think ahead.
When you get to this point think about bar codes, or really think about it now. Your package needs a bar code. Your box needs a bar code. Your outer box needs a bar code. It gets crazy.
When you start growing will you be able to keep up with all the paperwork? Will you need to hire a bookkeeper? Accountant? That's another expense to think about.
Hope this helps a bit.