The band was formed in 2001 by Toma Feterman, who has ancestors of Polish, Rumanian and Jewish Russian origins. A self taught multi-instrumentalist, Toma sings as well as playing the banjo and the trumpet.

After initially performing music that covers a wide range from punk to French “chanson”, he decides to gather a group to perform music from Eastern Europe – a way of returning to his roots. At the beginning, the musicians were friends that played traditional songs. He meets Olivier Llugany in January 2001. Llugs started playing the fiscorn (a traditional bass bugle from Catalunia) at a young age, in the Sardane orchestras of his region (coblas).
As a result, this unusual instrument has been introduced into the project – it has certain similarities with the tubas of the brass bands from the Balkans.

It is thanks to Llugs that the group has that particular brass sound. Between 2001 and 2003, the band becomes popular in Paris, acquiring a reputation as a celebratory gypsy-klezmer ensemble. They can be found on all the stages of the Paris musical underground scene.

La Caravane Passe plays several times a week in places like l’Olympic, Les Trois Arts, Le Café Montmartre… The group also participates quite regularly in cabaret evenings where they meet and accompany a big variety of circus acts. During 2002, La Caravane Passe creates, together with Môh Aroussi, « le Vrai Faux Mariage » (the Real Fake Wedding), a kind of gypsy rock cabaret performance.

In 2003 Ben Body, a great friend of Toma Feterman, becomes bass and double bass player for the band.
2004 is the great turning point in the group’s career : that year, Pat Gigon, Ben Body and Cyril « Zinzin » Moret join La Caravane Passe : the set-up is complete.