I've noticed that libertarians come in three flavors. There is some overlap on the boundaries, but it works for a hastily kludged together metaphor.
The first kind is the Practical group. They're the ones who support a smaller government because the government doesn't do a very good job at most of the things that it tries to do, and we'd be better off if they didn't try to do so much. They criticize the government as an inefficient solution to society's problems.
The second kind is the Ethical group. They're the ones who talk about how many government programs are unconstitutional, equate taxation with theft, and generally decry the government as having far exceeded its proper role as narrowly defined in the Constitution.
The third kind is the Crazy group. They're the ones who think that the government is mind controlling people with fluoride, that the moon landings were fake, and that a secret conspiracy of The Federal Reserve/Freemasons/Illuminati/Lizardmen is running the world from behind the scenes. Some people just fixate on one conspiracy, while others go all-in and believe everything.
The first and second kind get along fine, and a lot of libertarians use arguments from both camps. They're the mainstream libertarians, as much as the term can be applied to a somewhat fringe group, anyway. The problem, and where almost all of the intralibertarian conflict comes from, is the third group. The Crazies regularly cling to the notion that a small circle of elites could effectively run the world in secret, which is strictly antithetical to the Practical notion that it's just not possible for a few politicians to micromanage a large and complex society. The Crazies make the Ethical group uncomfortable because they both agree that the government is evil and doing things that it shouldn't, but the Ethicals think that the list includes things like the income tax and covert surveillance, while the Crazies think the list includes vaccinating children and pasteurizing milk.
My deepest wish, at least so far as this topic goes, is for the Crazies to just shut up. Seriously, guys, you are not helping. Every time some other group attacks libertarians, they go after you because you're the easiest targets and you make us all look bad. Every time there is a debate between the sane libertarians and someone else, you jump in and start lobbing your Crazy softball arguments that the other side just knocks out of the park because Lizardmen aren't real, fluoride doesn't allow mind control, Obama wasn't born in Kenya, and vaccination is settled science, you idiots. You make the rest of us look like fools because we have you rabid nutjobs on our side, screwing up the signal to noise ratio with your incoherent nonsense.
Listen, Crazies; if you really want to have any chance of the libertarian movement making any progress in the political arena, you need to stop making yourselves the insane poster-children of the libertarian movement. Just take your medication—I promise they aren't trying to contaminate your precious bodily fluids—and never talk about politics again.