A vanguard party and a certain separation between the party and the mass are made necessary precisely because of the inevitably inadequate level of class consciousness on the part of broad working masses. As Lenin repeatedly stressed, this is a complex dialectical relationship - a unity of separation and integration - which totally conforms to the historical peculiarities of the revolutionary struggle for a socialist revolution.
This separate party, however, originates within bourgeois society which, with its inherent features of a universal division of labour and commodity production, tends to bring about a reification of all human relations.  This means that the building of a party apparatus separated from the working masses involves the danger of this apparatus becoming autonomous. When this danger develops beyond an embryonic stage, the tendency arises for the self-preservation of the apparatus to become an end in itself, rather than a means to an end (successful proletarian class struggle).
This is the root of the degeneration of both the Second and the Third Internationals, i.e., the subordination of the mass social-democratic as well as the Communist parties of Western Europe to conservative, reformist bureaucracies which, in their day-to-day practice, have become part of the status quo. 
Bureaucracy in workers organisations is a product of the social division of labour, i.e., of the inability of the working masses, who are largely excluded from the cultural and theoretical process of production under capitalism, to themselves regularly take care of all the tasks which must be dealt with within the framework of their organisation. Attempts to do this anyway, as was often done at the onset of the workers movement, provide no solution because this division of labour completely corresponds to material conditions and is in no way invented by wicked careerists. If these conditions are overlooked, primitivism, ignorance and the brawling it produces will place the same limitations on the movement as would otherwise be set by the bureaucracy. Having taken a different point of departure here - that of organisational technique instead of the level of consciousness - we have run up against the same problem which we had already cleared up earlier: namely, that it would be giving the capitalist mode of production too much credit to assume it to be a perfect school for preparing the proletariat for independent activity, or that it automatically creates the ability of the working masses to spontaneously recognise and achieve all the objectives and organisational forms of their own liberation.
Lenin, in his first debate with the Mensheviks, very much underestimated the danger of the apparatus becoming autonomous and of the bureaucratisation of the workers parties. He proceeded from the assumption that the danger of opportunism in the modern labour movement was a threat coming mainly from petty-bourgeois academicians and petty-bourgeois "pure trade unionists." He made fun of the struggle of many of his comrades against the danger of "bureaucratism." Actually, history showed that the greatest source of opportunism in the Social Democracy before the first world war came from neither the academicians nor the "pure trade unionists" but from the social-democratic party bureaucracy itself i.e., from a practice of "legalism" limited on the one hand to electoral and parliamentary activity, and on the other to a struggle for immediate reforms of an economic and trade union nature. (To merely describe this practice is to confirm how much it resembles that of today’s West European Communist parties!)
Trotsky and Luxemburg recognised this danger more accurately and earlier than Lenin. As early as 1904 Luxemburg expressed the thought that a "difference between the eager attack of the mass and the [overly] prudent position of the Social Democracy" was possible.  The thought is hardly expressed before it is discarded; the only possible validity it might have would be in the imaginary case of an "overcentralization" of the party along Leninist lines. Two years later Trotsky already expresses this with more precision:
The European Socialist parties, particularly the largest of them, the German Social-Democratic Party, have developed their conservatism in proportion as the masses have embraced socialism and the more these masses have become organised and disciplined. As a consequence of this, Social Democracy as an organisation embodying the political experience of the proletariat may at a certain moment become a direct obstacle to open conflict between the workers and bourgeois reaction. In other words, the propagandist-socialist conservatism of the proletarian parties may at a certain moment hold back the direct struggle of the proletariat for power. 
This prognosis has been tragically confirmed by history. Lenin did not yet see this until the eve of the first world war, whereas the German left had long before shed its illusions about the social-democratic party administration.