We should not associate with each and every devotee. We should be very careful with whom we associate. Most worshiped Srimad Rupa Goswami has mentioned – how we should associate (Sri Upadesamrita, verse 5):

kṛṣṇeti yasya giri taṁ manasādriyeta
dīkṣāsti cet praṇatibhiś ca bhajantam īśam
śuśrūṣayā bhajana-vijñam ananyam anya-
nindādi-śūnya-hṛdam īpsita-saṅga-labdhyā

One should mentally offer respects to that person who chants the Holy Name of Krishna. One should bow down unto that devotee who has accepted initiation and is engaged in worship of the Lord, and one should desire the association of those pure devotees who are advanced in bhajana and whose hearts are totally devoid of blasphemy to others and faithfully render service unto them.

“We must associate with only a devotee who follows the same path of bhajan as we (sajatiya), serene, more advanced than us, one who does not indulge in ninda (aninduk), and learned in bhajan.”
(Srila Vishwanath Chakravarti)

If we do not follow these rules of association, our bhajan may suffer and more harm will be done than anything good. Harikatha alone does not necessarily mean ‘sadhu-sanga’. If both the devotees quarrel on some point and both get agitated, then such association will result in Vaishnava-aparadha. If both the devotees do not follow the same path of bhajan, then each will try to justify his own path, or both may try their level best to prove that his bhajan is greater than the other’s. This will result in competitiveness and ill feeling.

If the devotee is not learned in bhajan, he may give wrong instruction due to his lack of knowledge. Ultimately, we will suffer. If the devotee, with whom we associate, indulges in ninda, then he may criticize some Vaishnava while giving some example during Harikatha. Hearing such ninda will harm our bhajan. Hence, we must be extremely careful while associating with devotees. Of course, we should pay respectful obeisance to all, but we must not associate with everybody.
(Srila Binod Bihari das Babaji)

—

After taking shelter of a bonafide guru, one gets instructions about the various rules and regulations of the devotional process (sadācāra). Finally, he gets the fortune to associate with soft-hearted realized sādhus endowed with a similar mood of devotion. When a sādhaka is thus blessed with a life full of devotion, he should, after taking shelter of the feet of Śrī Guru, certainly associate with bonafide mahā bhāgavata Vaiṣṇavas. Their example should be the framework for one’s devotional life. Serving them and associating with them makes the sādhaka’s attachment to bhajana firm.

Śrīmat Rūpa Gosvāmī has said svajātīyāśaye snigdhe sādhau saṅgaḥ svato vare – There are different moods in devotion as aiśvarya, mādhurya, dāsya, sakhya, and so forth. A devotee should associate with the sādhus having a similar mood as him. Moreover, that sādhu should be more advanced and realized than oneself and should be always affectionate due to his soft-hearted merciful nature. The association of such a mahā-bhāgavata shows great fortune on a sādhaka devotee. A devotee can easily relish the mellow of devotion by seeing him, touching him, speaking with him, and hearing hari-kathā from him. The Guru therefore instructs his disciple to associate with such sādhus, so that he can become blessed.

—

By constantly performing bhajana in the association of the sadhus one gets purified and as a concomitance all ignorance will be destroyed.
In Srimad Bhagavata (11.26.31) it is seen: yathopasrayamanasya bhagavantam vibhavasum; sitam bhayam tamo’pyeti sadhun samsevatas tatha – “Just as cold, fear and darkness disappear for a man who takes shelter of the firegod, so the fear of ignorance and rebirth due to fruitive activities disappears from a man who serves the saints.”

Srila Visvanatha Cakravartipada has written the following commentary on this verse: sviyaudana siddhyartham upasrayamanasya apyeti nasyati. tathaiva bhajana siddhyartham sadhun samsevyamanasya karmadi jadyam samsara bhayam bhajana vighnas ca – “When one kindles the fire to cook food, then fear, cold and darkness disappear as a concomitance. Similarly, by serving the saints one attains the perfection of prema, and as a concomitant factor one easily becomes free from fear of karma and material existence, and conquers over all obstacles to spiritual advancement.”

What’s more, association with saints is direct devotion, for it subdues the Lord Himself.

na rodhayati mam yogo na sankhyam dharma eva ca
na svadhyayas tapas tyago neshtapurtam na dakshina
vratani yajnas chandamsi tirthani niyama yamah
yathavaruddhe sat sangah sarva sangapaho hi mam
(Bhag. 11.12.1-2)

The Lord told Sri Uddhava: “O Yadunandana! You are My dearmost friend and servant, therefore I will tell you a great secret, listen! ashtanga yoga, sankhya (discrimination of the truth), religious principles like non-violence or adherence to Varnasrama Dharma, recitation of the Vedas, penances, sannyasa, performance of fire-sacrifices, making gardens, digging wells, giving charity, performing vows, worshipping the demigods, reciting mantras, visiting the holy places of pilgrimage, following regulative principles or performing Homa-sacrifices – all these things cannot subdue Me as much as associating with sadhus, giving up all other association!”

Hence Srimad Bhagavata says:

ayam hi paramo labho nrinam sadhu samagamah
(Bhag. 12.10.7)

“Association with saints is the highest attainment for everyone”.

In this way the indescribable glories of associating with saints is repeatedly described in the transcendental scriptures.

sadhu sanga sadhu sanga sarva sastre koy;
lava matra sadhu sange sarva siddhi hoy (C.C.)

“Associate with the saints! Associate with the saints! This is what all the scriptures say. By associating with a saint for only a moment all perfection is attained!”
(Śrī Rādhākuṇḍa Mahānta Paṇḍita Śrī Ananta Dās Bābājī Mahārāja)