At my wits end with my newborn!?

My baby is 3 1/2 weeks old. He has been formula fed since birth because I'm taking an anti-seizure medication that pes through the milk. He was fantastic for the first week - mostly sleeping and waking every three hours or so to feed. Shortly after that first week he became very fussy at nighttime, and nights have slowly turned into a nightmare. We tried switching his formula to Gentlease (for gy and fussy babies) and that did not help. We also started giving him Gripe water but read online it can actually make the problem worse because it has baking soda in it, so we ordered an organic product online called colic calm and are waiting for it to arrive. The problem is, I'm not convinced it's colic. He's not inconsolable, if you hold him in your arms certain ways he'll stop crying, and it's only at night - during the day we can lay him down in his crib and he'll sleep no problem. But once it's time for us to go to bed, we feed him and lay him down in his crib, he'll fall asleep for 10 min or so, and just as we're drifting off to sleep he starts crying and won't stop until you come pick him up. Sometimes he'll continue crying even after you pick him up (like he has a tummy ache or something), other times he'll stop crying as soon as you pick him up. We'll get him to fall asleep in our arms, and we'll even wait 10 or 15 min sometimes for him to fall into a deep sleep before we try laying him down. The end result is always the same - he'll either cry as soon as you lay him down, or he'll fall back asleep only to start crying 10 or 15 min later and not stop until you pick him up. The kicker - he will sleep in his swing (we have a cradle swing for him) no matter what, no problem. We don't want him sleeping in the cradle swing always because it seems like it isn't safe as a sleeping arrangement (pillow under and on either side of his head, etc.). Does anyone have ANY suggestions?? We're losing our mind getting absolutely no sleep.