Personal computing discussed

rsync always copies certain files with --delete

Moderators: renee, SecretSquirrel, notfred

Post Reply
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:
Contact Crayon Shin Chan

rsync always copies certain files with --delete

Fri May 07, 2010 10:36 pm

A rsync server is running on my Windows 7 laptop, which has foobar2000. Each time I rsync the music folders on my laptop and my Mac (running OS X 10.5) with --delete, I notice that it always recopies certain files, like certain tracks within an album. Without the --delete option, (rsync -av), the problem doesn't come up.
Normally I'd think that it's a character set problem, but I used foobar2000 (which btw advertises full UTF-8 support) to rename all my files to a consistent naming scheme, so that can't possibly be the problem. Plus, some files with Chinese/Japanese/Korean filenames aren't affected.
Any ideas?
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
Top
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: rsync always copies certain files with --delete

Fri May 07, 2010 10:49 pm

Could still be some sort of strange character set/encoding issue... maybe. Are there any characters in these files' names that aren't in the names of the files that don't exhibit this misbehavior?

Are you specifying the --times option (to sync the file modification times)?

The --modify-window option can sometimes be helpful when syncing between OSes or filesystems which have different file timestamp granularities.
Nostalgia isn't what it used to be.
Top
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:
Contact Crayon Shin Chan

Re: rsync always copies certain files with --delete

Fri May 07, 2010 11:04 pm

just brew it! wrote:
Could still be some sort of strange character set/encoding issue... maybe. Are there any characters in these files' names that aren't in the names of the files that don't exhibit this misbehavior?

Are you specifying the --times option (to sync the file modification times)?

The --modify-window option can sometimes be helpful when syncing between OSes or filesystems which have different file timestamp granularities.

Yes, now that you mention it, it might be a weird charset/encoding issue. I'm not using --times, the command line is:

Code: Select all
sudo rsync -av --delete laptop::music ~/Music



I made foobar2000 rewrite the filenames, but the problem still occurs. For instance:
Code: Select all
Epik High/Remapping the Human Soul - Part 2 - The Heart/
deleting Nightmare - GIANIZM/10. ジャイアニズム天.mp3
deleting Nightmare - GIANIZM/09. ジャイアニズム究.mp3
deleting Nightmare - GIANIZM/08. ジャイアニズム罰.mp3
deleting Nightmare - GIANIZM/07. ジャイアニズム叱.mp3
deleting Nightmare - GIANIZM/06. ジャイアニズム碌.mp3
deleting Nightmare - GIANIZM/05. ジャイアニズム誤.mp3
deleting Nightmare - GIANIZM/04. ジャイアニズム死.mp3
deleting Nightmare - GIANIZM/03. ジャイアニズム惨.mp3
deleting Nightmare - GIANIZM/02. ジャイアニズム痛.mp3
Epik High/Swan Songs/


but track number 1 of GIANIZM also has a Japanese filename:

Code: Select all
[shinichi@motoko:~/Music/Nightmare - GIANIZM] ls -lh
total 183680
-rwxr-xr--  1 shinichi  staff    12M May  6 15:50 01. 自傷(少年テロリスト).mp3
-rwxr-xr--  1 shinichi  staff   9.2M May  6 15:50 02. ジャイアニズム痛.mp3
-rwxr-xr--  1 shinichi  staff   6.5M May  6 15:50 03. ジャイアニズム惨.mp3
-rwxr-xr--  1 shinichi  staff   9.6M May  6 15:50 04. ジャイアニズム死.mp3
-rwxr-xr--  1 shinichi  staff   5.6M May  6 15:50 05. ジャイアニズム誤.mp3
-rwxr-xr--  1 shinichi  staff    10M May  6 15:50 06. ジャイアニズム碌.mp3
-rwxr-xr--  1 shinichi  staff    12M May  6 15:50 07. ジャイアニズム叱.mp3
-rwxr-xr--  1 shinichi  staff   7.3M May  6 15:50 08. ジャイアニズム罰.mp3
-rwxr-xr--  1 shinichi  staff   6.1M May  6 15:50 09. ジャイアニズム究.mp3
-rwxr-xr--  1 shinichi  staff    11M May  6 15:50 10. ジャイアニズム天.mp3
-rwxr-xr--  1 shinichi  staff    88K May  5 18:46 VPCC-80646.jpg
-rwxr-xr--  1 shinichi  staff    94K May  5 18:21 VPCC-81654.jpg


Code: Select all
deleting Ravel - Complete Solo Piano Works I/18 - Pavane pour une infante défunte (Assez doux, mais d'une sonorité large).flac
deleting Ravel - Complete Solo Piano Works I/13 - Valses nobles et sentimentales - IV. Assez animé.flac
deleting Ravel - Complete Solo Piano Works I/12 - Valses nobles et sentimentales - III. Modéré.flac
deleting Ravel - Complete Solo Piano Works I/10 - Valses nobles et sentimentales -I- Modére - trés franc.flac
deleting Ravel - Complete Solo Piano Works I/08 - Sonatine - III. Animé.flac
deleting Ravel - Complete Solo Piano Works I/06 - Sonatine - I. Modéré.flac
deleting Ravel - Complete Solo Piano Works I/05 - Miroirs - V La vallée des cloches - Très lent.flac
deleting Ravel - Complete Solo Piano Works I/03 - Miroirs - III Une barque sur l'océan - D'un rythme souple.flac
deleting Ravel - Complete Solo Piano Works I/02 - Miroirs - II Oiseaux tristes - Très lent.flac
Ghost in the Shell/Ghost in the Shell SAC OST2/
deleting Ravel - Complete Solo Piano Works I/01 - Miroirs - I Noctuelles - Très léger.flac
Ghost in the Shell/Ghost in the Shell SAC OST2/Scans/


Ravel track number 4 was not copied, apparently because it doesn't have the character é:
Code: Select all
[shinichi@motoko:~/Music/Ravel - Complete Solo Piano Works I] ls -lh
total 470440
-rwxr-xr--  1 shinichi  staff    17M Feb  2 11:57 01 - Miroirs - I Noctuelles - Très léger.flac
-rwxr-xr--  1 shinichi  staff    12M Feb  2 11:57 02 - Miroirs - II Oiseaux tristes - Très lent.flac
-rwxr-xr--  1 shinichi  staff    26M Feb  2 11:57 03 - Miroirs - III Une barque sur l'océan - D'un rythme souple.flac
-rwxr-xr--  1 shinichi  staff    25M Feb  2 11:57 04 - Miroirs - IV- Alborada Del Gracioso (Assez Vif).flac
-rwxr-xr--  1 shinichi  staff    17M Feb  2 11:57 05 - Miroirs - V La vallée des cloches - Très lent.flac
-rwxr-xr--  1 shinichi  staff    15M Feb  2 11:57 06 - Sonatine - I. Modéré.flac
-rwxr-xr--  1 shinichi  staff    11M Feb  2 11:57 07 - Sonatine - II. Mouvement de menuet.flac
-rwxr-xr--  1 shinichi  staff    15M Feb  2 11:57 08 - Sonatine - III. Animé.flac
-rwxr-xr--  1 shinichi  staff    23M Feb  2 11:57 09 - Menuet antique.flac
-rwxr-xr--  1 shinichi  staff   4.8M Feb  2 11:57 10 - Valses nobles et sentimentales -I- Modére - trés franc.flac
-rwxr-xr--  1 shinichi  staff   7.1M Feb  2 11:57 11 - Valses nobles et sentimentales - II. Assez lent.flac
-rwxr-xr--  1 shinichi  staff   4.7M Feb  2 11:57 12 - Valses nobles et sentimentales - III. Modéré.flac
-rwxr-xr--  1 shinichi  staff   3.6M Feb  2 11:57 13 - Valses nobles et sentimentales - IV. Assez animé.flac
-rwxr-xr--  1 shinichi  staff   4.3M Feb  2 11:57 14 - Valses nobles et sentimentales - V. Presque lent.flac
-rwxr-xr--  1 shinichi  staff   3.0M Feb  2 11:57 15 - Valses nobles et sentimentales - VI. Vif.flac
-rwxr-xr--  1 shinichi  staff   9.4M Feb  2 11:57 16 - Valses nobles et sentimentales - VII. Moins vif.flac
-rwxr-xr--  1 shinichi  staff    12M Feb  2 11:57 17 - Valses nobles et sentimentales - VIII. Epilogue- Lent.flac
-rwxr-xr--  1 shinichi  staff    20M Feb  2 11:57 18 - Pavane pour une infante défunte (Assez doux, mais d'une sonorité large).flac
-rwxr-xr--  1 shinichi  staff   6.2K Feb  2 11:30 ravel1.log


These are just some of the files that are affected - as you can see, it's so diverse that any attempt to find specific characters that don't affect this would be quite daunting:
Code: Select all
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/10 - 大利根無情.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/09 - 夜霧のブルース.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/08 - 黄昏のビギン.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/07 - 旅の夜風.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/06 - 東京の屋根の下.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/05 - かえり船.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/04 - 長崎の鐘.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/03 - 皆の衆.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/02 - 若いお巡りさん.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/01 - 憧れのハワイ航路.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD2/
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/09 - あばよ（アルバムバージョン）.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/08 - きよしのソーラン節.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/07 - 陽春.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/06 - 朝顔日記.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/05 - 玄武.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/04 - 希望という名の最終列車.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/03 - きよしの一心太助.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/02 - 扇.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/01 - あばよ.mp3
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/CD1/
deleting 演歌名曲コレクション7～あばよ・きよしのソーラン節～/
deleting キリンジ - スウィートソウル/12 - クレイジー・サマー(instrumental).mp3
deleting キリンジ - スウィートソウル/11 - 悪い習慣(instrumental).mp3
deleting キリンジ - スウィートソウル/10 - ザ・チャンス(instrumental).mp3
deleting キリンジ - スウィートソウル/09 - 愛のCoda(instrumental).mp3
deleting キリンジ - スウィートソウル/08 - ブラインドタッチの織姫(instrumental).mp3
deleting キリンジ - スウィートソウル/07 - スウィートソウル(instrumental).mp3
deleting キリンジ - スウィートソウル/06 - クレイジー・サマー.mp3
deleting キリンジ - スウィートソウル/05 - 悪い習慣.mp3
deleting キリンジ - スウィートソウル/04 - ザ・チャンス.mp3
deleting キリンジ - スウィートソウル/03 - 愛のCoda.mp3
deleting キリンジ - スウィートソウル/02 - ブラインドタッチの織姫.mp3
deleting キリンジ - スウィートソウル/01 - スウィートソウル.mp3
deleting キリンジ - スウィートソウル/
deleting Assorted/쿨론 - 쿵따리샤바라.MP3
deleting Assorted/지오디 - 어머님께.mp3
./
12Riven - the Ψcliminal of integral OST/
12Riven - the Ψcliminal of integral OST/CD1/
12Riven - the Ψcliminal of integral OST/CD2/
deleting Epik High/Epik High - Remixing The Human Soul/18. 전자깡패 (by 삼자돼면) (Hidden Track).flac
deleting Epik High/Epik High - Remixing The Human Soul/05. 1분 1초, A Little Memory (feat. Taru).flac
deleting Epik High/Epik High - Remixing The Human Soul/04. 버려진 우산 (feat. Lisa).flac
deleting Epik High/Epik High - [e]/CD1/12 - Happy Birthday To Me (feat. 하동균).mp3
deleting Epik High/Epik High - [e]/CD1/11 - Breathe (Mithra's Word) (feat. 한희정).mp3
deleting Epik High/Epik High - [e]/CD1/07 - 트로트.mp3
deleting Epik High/Epik High - [e]/CD1/06 - 통기타 (Skit).mp3
deleting Epik High/Epik High - [e]/CD1/03 - 선물 (feat. 박지윤).mp3
deleting Epik High/Epik High - [e]/CD2/13 - 흉 (feat. MYK, YDG, Dok2).mp3
deleting Epik High/Epik High - [e]/CD2/12 - High Skool Dropout (반항하지 마).mp3
deleting Epik High/Epik High - [e]/CD2/07 - 말로맨.mp3
deleting Epik High/Epik High - [e]/CD2/04 - 따라해 (Wannabe) (feat. Mellow).mp3
deleting Epik High/High Society/17 - 신사들의 몰락 (Good Evening).mp3
deleting Epik High/High Society/16 - 뒷담화.mp3
deleting Epik High/High Society/14 - Daydream (사직서).mp3
deleting Epik High/High Society/13 - 혼자라도 (feat. Clazziquai).mp3
deleting Epik High/High Society/12 - 뚜뚜루.mp3
deleting Epik High/High Society/11 - 11월 1일 (ft. 김재석 of WANTED.mp3
deleting Epik High/High Society/10 - 피해망상 Part 3 (feat. TBNY).mp3
deleting Epik High/High Society/08 - 신사들의 절약정신 (Good Aftern.mp3
deleting Epik High/High Society/06 - My Ghetto (feat. 김연우).mp3
deleting Epik High/High Society/03 - 평화의 날.mp3
deleting Epik High/High Society/01 - 신사들의 산책 (Good Morning).mp3
deleting Epik High/Pieces, Part One/16 - 당신의 조각들 (feat. 지선).mp3
deleting Epik High/Pieces, Part One/15 - 우산 (feat. 윤하).mp3
deleting Epik High/Pieces, Part One/14 - 낙화 (落花).mp3
deleting Epik High/Pieces, Part One/10 - Ignition (feat. 나윤권).mp3
deleting Epik High/Pieces, Part One/06 - Girl (feat. 진보).mp3
deleting Epik High/Pieces, Part One/05 - 연필깎이 (feat. Kebee).mp3
deleting Epik High/Pieces, Part One/04 - One (feat. 지선).mp3
deleting Epik High/Pieces, Part One/03 - 서울 1시13 AM (Short Piece).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/13 - In Peace (연주곡).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/12 - 혼 (Soul).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/10 - 희생양 (feat. Sweet Sorrow).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/09 - 피해망상 pt.1 (feat. Junggigo).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/07 - Exile (연주곡).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/04 - 실어증 (feat. Paloalto).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/03 - 알고 보니 (feat. Jinbo).mp3
deleting Epik High/Remapping the Human Soul - Part 1 - The Brain/02 - 白夜 (백야 - White Night).mp3
deleting Epik High/Remapping the Human Soul - Part 2 - The Heart/13 - 해복합니다 (feat. JW of Nell).mp3
deleting Epik High/Remapping the Human Soul - Part 2 - The Heart/07 - 중독 (feat. Wanted).mp3
deleting Epik High/Remapping the Human Soul - Part 2 - The Heart/06 - 선곡표 (feat. DJ Zio).mp3
deleting Epik High/Remapping the Human Soul - Part 2 - The Heart/05 - 거미줄 (feat. Itta).mp3
deleting Epik High/Swan Songs/12 - 도시가 눈을 감지 않는 이유.mp3
deleting Epik High/Swan Songs/08 - 이별, 만남... 그 중점에서.mp3
deleting Epik High/Swan Songs/06 - 그녀는 몰라.mp3

I replaced the cygwin1.dll in C:\Program Files\cwrsync\bin with a Unicode aware version from okisoft.co.jp, restarted the server, and it still occurs.

Worth noting though is that it seems Chinese filenames aren't affected at all!
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
Top
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:
Contact Crayon Shin Chan

Re: rsync always copies certain files with --delete

Fri May 07, 2010 11:11 pm

Since Chinese filenames weren't affected, I used foobar2000 to rename them twice - once to a custom scheme, and then to the original naming scheme. The Chinese files are STILL not affected. Very weird, it would have meant foobar2000 was the problem if they were affected.
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
Top
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: rsync always copies certain files with --delete

Fri May 07, 2010 11:22 pm

It is probably worth trying the --times and/or --modify-window options. It might not help, but shouldn't hurt either.
Nostalgia isn't what it used to be.
Top
 
bthylafh
Maximum Gerbil
Posts: 4320
Joined: Mon Dec 29, 2003 11:55 pm
Location: Southwest Missouri, USA

Re: rsync always copies certain files with --delete

Fri May 07, 2010 11:23 pm

Have the latest version of cwrsync? Looks like the current rev uses Cygwin v1.7 DLL, which should be natively UTF-8 aware, and maybe this is a bug with the version you've got.

See here for how to set up Cygwin's locale: http://cygwin.com/cygwin-ug-net/setup-locale.html

edit: that DLL from okisoft.co.jp is for Cygwin 1.5, which is deprecated.
Hakkaa päälle!
i7-8700K|Asus Z-370 Pro|32GB DDR4|Asus Radeon RX-580|Samsung 960 EVO 1TB|1988 Model M||Logitech MX 518 & F310|Samsung C24FG70|Dell 2209WA|ATH-M50x
Top
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On