rePear User's Guide

Author: Martin J. Fiedler <martin.fiedler@gmx.net>
Version: 0.4.1
Date: 2009-02-25

For the impatient ...

Initial Setup

1. initialize the iPod with iTunes
2. remove all tracks from the iPod with iTunes (optional)
3. install rePear on the iPod
   • On Windows: unpack the .zip file into the iPod's root directory
   • On other systems: copy rePear (the .py files) into the iPod's root directory
4. run repear config or use the GUI (rePearUI.exe) to configure rePear
5. run repear dissect or click the »dissect« button in the GUI to have the existing iTunes music library broken down into files and folders with an /artist/album/title scheme (optional, dangerous)
6. copy music onto the iPod with the file manager you like best
7. run rePear to freeze the database
8. wait
9. safely(!) disconnect the iPod
10. listen to your music (optional, but recommended :)

Regular Maintenance (i.e. Manage your Music)

1. connect the iPod to your computer
2. run rePear to unfreeze the database
3. do whatever you want with your music files
4. run rePear to freeze the database again
5. safely(!) disconnect the iPod
6. enjoy your music

rePear Setup – Step by Step

First, hold your breath. This section is awfully long, but that's only because it's written in a very detailed manner. It should not take you longer than about half an hour to get everything going (or ten minutes if you already have a decent Python installation or are working on Windows).

Now, this is what you have to do to use rePear with your iPod:

1. Prepare the iPod

If your iPod is new, you have to connect it to your computer and have it initialized by iTunes at least one time. If you used your iPod with iTunes before, then it's up to you: You may delete all tracks from it to start all over with rePear, but you may as well keep the tracks installed by iTunes. If they contain correct ID3 tags and are all unencrypted MP3 or AAC, this should work fine.

You also have to find out where the iPod's root directory is. On Windows, this is the root directory of the iPod's drive letter (e.g. L:\). On Mac OS X, it's something like /Volumes/My iPod. On other Unixes, it depends – on my Linux box, it is usually be mounted at /media/sda2, for example.

Once you figured out where you can find the root directory, you should make a backup of the directory /iPod_Control/iTunes. rePear will automatically create a backup of the most important file in there, iTunesDB, but it is always better to have a second safety net. On Windows, it may be necessary to configure Explorer so that it shows hidden files and directories.

2. Install rePear

If you are on a Windows box, download the .zip file version of rePear and extract it into the iPod's root directory. You should get a rePear subdirectory and some files in the root directory. The two most importand of these are repear.bat, which is a small launcher script if you want to run the command-line version of rePear, and rePearUI.exe, the graphical user interface. This is all you need, everything else is already included in the ZIP file.

On UNIX-like operating systems, download the .tar.gz version and put all the .py and .ini files from the downloaded archive into the iPod root directory (this is a recommendation; if you don't put rePear there, you will have to use the –r switch described below).
You will also need a Python runtime to use rePear. On most Unix distributions (including OS X), it is installed by default. If it isn't, use the package manager to install it or look at the official download page to see what to do. Every version of Python between 2.3 and 2.6 should work.
If you intend to make use of rePear's Ogg Vorbis transcoding feature, you also need to install OggDec (from the vorbis-tools package) and LAME. On Unix-like systems, a vorbis-tools package should be available from your package manager, but you may have to compile and install LAME for yourself.
Furthermore, the cover artwork features require the Python Imaging Library (PIL). Again, this should be available through the package manager.

The »direct« way to start the command-line version rePear is to open a console window (on Windows: Start Menu, Run, cmd), go to the iPod root directory (e.g. with L: or cd "/Volumes/My iPod") and run repear [options here] (Windows) or python repear.py [options here] (Unix). This method is needed if you want to specify options or tell rePear which action to perform. If you don't need to do this (and you won't for most of the time), you can as well double-click repear.bat on Windows or use the graphical user interface, which has buttons for the most commonly used operations. On Mac OS X, after double-clicking repear.py, the system will ask whether to execute the program in the Terminal when you double-click it the first time. Confirm this message box. If it tells you that »no default application is specified«, click »Choose Application« and select »Terminal«.

3. Configure rePear

Some of rePear's features require a moderate amount of configuration – namely, the exact iPod model must be known to generate cover artwork and to use the last.fm scrobbling interface, you need to specify your last.fm login data.

These configuration steps can be done on the command line by running repear config. This will start a small text-based wizard that asks you to pick your model from a list and specify your last.fm credentials. If you don't use last.fm or don't want to use that feature, just press ENTER when asked for the login name.

In the Windows GUI, there's a »config« button that opens a small dialog window with the same options: a drop-down menu for the model selection and two fields for the last.fm login data. Just make the appropriate settings here and click »OK«.

Another very important step to follow is synchronizing the clocks of the iPod and the computer: If you intend to use last.fm scrobbling, a precise record of when which track has beed played is essential. So please double-check that the iPod's and the computer's clock agree and are set to the same timezome. This is important, because last.fm might reject your scrobbled tracks if there are any inconsistencies in the play times.

4. Import tracks (optional)

If you still have some tracks in the iPod database that were previously installed with iTunes, you can either keep them in place or have rePear sort them back into directories. If your iPod is empty, just skip this step completely.

If you choose to keep them inside the iPod's private directory structures, it will be fine. But note that you will not be able to modify them any further (not even delete them) without iTunes.

The other option is to have rePear dissect the iPod's music database. To do this, start rePear with the dissect parameter (like in repear dissect) or click the »dissect« button in the GUI. After asking you to confirm your command, rePear will create a new folder Dissected Tracks on the iPod. Inside this folder, rePear will create a typical artist/album/track directory structure. For example, you will get files like /Dissected Tracks/Some Artist/Nice Album/07 – Some Track.mp3. Tracks with album information missing will be put directly in the artist's folder. Tracks without even an artist name will be put directly into Dissected Tracks.

5. Copy your tracks onto the iPod

Now it's time to make the USB port glow! You can now copy, delete, move or rename tracks on your iPod. You can create virtually any directory structure you want. It's up to you how to organize the files. In particular, you don't need to put them in any special location. iTunes stores them with cryptic names in /iPod_Control/Music, but you don't need to do that. In fact, it's stronly suggested not to put your self-managed files here – just create a folder elsewhere on the iPod. For example, all my music files go directly into /Music.

6. Run rePear

Now it's time to run rePear without parameters or select the highlighted »freeze« option in the GUI. It will do what I refer to as freezing the iPod's database: It will scan the whole filesystem of the iPod for MP3, Ogg and AAC files and move them into the hidden area where the playable tracks are kept, transcoding Ogg to MP3 as needed. While doing this, rePear analyzes each file to get the ID3 metadata information to finally generate a new music database on the iPod.

I'm afraid that this process is quite a lengthy one – rePear has a »metadata cache« that remembers all information about the tracks so that they don't need to be re-scanned the next time when new files are added, but moving the files is very time-consuming in itself. You should make sure to enable the operating system's write cache to get best performance. As a rule of thumb, freezing a 8 GB flash-based iPod should take 2 to 5 minutes.

7. Disconnect the iPod and listen to your music

After the freezing process is complete, you can disconnect the iPod. You should always do this in a safe manner (»safe removal of devices« in the Taskbar Notification Area on Windows; the eject icon next to the iPod volume's icon in Finder on OS X; umount on other Unixes).

You should now be able to listen to your music. If not, please send me a bug report with a precise description of what exactly went wrong, including the log file generated by rePear and a (compressed!) copy of the /iPod_Control/iTunes/iTunesDB file in both versions: The one created by rePear and the one from your previous backup. (And now guess what that backup was good for ... not only for sending me bug reports, but also for fixing your broken iPod again!)

Managing the rePear tracks – Step by Step

Let's imagine that everything went OK (and I hope it did!) and now you want to put new music onto the iPod, remove old tracks or do other maintaining tasks. Here is what you have to do:

1. Plug in the iPod and run rePear

If you start rePear without any parameters, it will remember that the databaze is frozen and automatically do the right thing: unfreeze it. The GUI doesn't do this completely automatically, but at least it will recognize that »unfreeze« is the most logical thing to do and thus, it will highlight the according button.

Unfreezing means that all the files you put on the iPod that mysteriously disappeared while freezing (because they were moved to the iPod's sected music hideout) will be put back again. This will not take as long as freezing it, but on iPods that are slow with I/O (most newer models), it will be close.

2. Manage your Music

If there were no errors, all the files will be back at their original locations and you can now start to copy, move and rename files or directories at will.

3. Run rePear again

Before disconnecting the iPod, you have to run rePear once again so that it can freeze the iPod's music database and move the music files away again. All files that you left untouched will be processed in an instant; new or modified files will take a little bit longer to process.

If you enabled the last.fm scrobbling feature, this will also be the point where the information about when you listened to which tracks will be submitted to last.fm. If a track has been played more than once since the last freeze or update, only the last time it has been played will be submitted. This isn't rePear's fault – the iPod simply only saves the last playback time.
If any error occurs while uploading the information to last.fm (like a broken network connection, or just because you're offline), the information will not be lost. Instead, it will be stored and re-submitted the next time the freeze or update action is executed.

4. Disconnect the iPod and listen to your music

Just like in the setup procedure described above, you may now safely (I mean it!) disconnect the iPod and enjoy your music.

Details on playlists

There are two ways to generate playlists with rePear: You can put normal M3U playlist files anywhere on the iPod and rePear will add them to the Playlist menu. These playlists will have the same name as the .m3u file they're created from. The directory name won't be included, so please make sure you don't have a dozen .m3u files with same names on your iPod.

The other, more advanced method of playlist generation are »automatic« playlists that are created from the main database with some user-specified rules. In the default configuration, there's already one such playlist: »Hot New Stuff« is where rePear puts all new and changed tracks it found into. If you don't like that, you can disable that, of course. This chapter describes how to do that.

The master playlist file

All automatic playlists and other playlist options are specified in a single file, which is repear_playlists.ini in the root directory of the iPod. If this file is not present, rePear assumes default options and doesn't generate any automatic playlists (not even »Hot New Stuff«).

The master playlist file looks like a normal Windows INI file, which means it is a text file with lines in the »key = value« format. It is subdivided into sections that start with a line with the section name in square brackets, like »[this]«. Comments start with a semicolon (;).

Each section in the master playlist file describes one playlist. The playlist name is derived from the section title. There's one exception, though: The first part of the file, before the first section header, is used for general, global playlist options.

Some options require boolean values. In this case, you can use either 1/0, yes/no, y/n, on/off, true/false or enable/disable to specify whether that option shall be used or not.

Global playlist options

The following options are available in the global options (pseudo-)section at the beginning of the repear_playlists.ini file:

skip album playlists (boolean, default: enabled)

This option specifies whether playlists that cover exactly one album will be included or not. Normally, these playlists are pointless: The album appears under »Albums«, there's no reason why it should be under »Playlists«, too. This means that you can keep the .m3u files that usually come with album downloads, without having them clutter your Playlists menu on the iPod. That's why this option is enabled by default – if you don't like it, you can disable it, though.

directory playlists (boolean, default: disabled)

If this option is enabled, rePear will create a playlist for every folder on the iPod filesystem it finds playable files in. Note that the playlist name will only contain the last component of the path name. The files in /Music/foo/bar/*.mp3 go into a playlist called »bar«, for example.

Automatic playlist options

The following options can be used to specify automatic playlists:

include = <path>

Specifies which files shall be included in the playlist, either using a filename pattern like »*.mp3« or a directory name like »/Music«. In the latter case, the specified directory and all subdirectories will be included.
By default, no files or directories are included in a playlist. This means that a playlist without any include (or new or changed) statements will be empty. So, to make a usable playlists, there needs to be at least one include statement. The number of includes per playlists is unlimited, so it's perfectly possible to include multiple directories.

exclude = <path>

This works like include, but it specifies which files shall not be included in the playlist. exclude is »stronger« than include, so you can exclude subdirectories of other directories. The number of excludes per playlists is also unlimited.

new = <boolean>

If this option is enabled, the playlist will contain all files that reTune found that were not present during the last freeze operation. Note that new is even stronger than include or exclude.

changed = <boolean>

If this option is enabled, the playlist will contain all files whose metadata changed since the last freeze operation. Like new, changed is even »stronger« than include or exclude.

shuffle = <mode>

Selects whether or not the playlist shall be shuffled, and which shuffle algorithm shall be used. The default (0, no, off, false, disabled or none) will keep the tracks in their original order. When this option is enabled (using 1, yes, on, true, enabled or balanced), an advanced shuffle algorithm will be used that creates a not completely random, but very homogenous order of the tracks. The detailed algorithm is described on this web page. Alternatively, a normal random shuffle can be selected with 2, random or standard.

sort = <criteria>

This option specifies the criteria after which the tracks in the playlist shall be sorted. For a detailed explanation of the syntax of these criteria, read below. If there is neither a sort nor a shuffle statement in a playlist definition, the files will be sorted by path and filename. If sorting is enabled, it will take place after shuffling. This means that sorting is »stronger« than shuffling. If there are multiple sort options in a playlist definition, the sort operations will be performed in the same order as defined, so the last sort will be the strongest one.

Sort criteria

The following sort criteria are defined:

Multiple criteria can be combined with commas to form complex sort orders. For example »artist, year, album, disc number, track number, title« will sort the tracks by artist, the tracks of each artist will then be sorted by year, the tracks of each year will then be sorted by album, disc number, track number and finally by title. Note that sorting text values will always take place in a case-insensitive manner.

There are also various modifies for each subcriterion which are added as prefixes: A plus sign (»+«) will sort in ascending order, which is also the default. A minus sign (»-«) will sort in descending order. Angle brackets specify where tracks will be sorted that are missing the value associated with the sort criterion: With »<«, empty values will be sorted to the front, with »>« (the default), empty values will be sorted to the back of the list. The ordering of the empty values is always independent from the main sort order. »-year«, for example, will sort the tracks in descending order by time, but tracks that are missing the »year« metadata field will still appear at the end of the sorted list. If this is not desired, »<-year« needs to be written.

Playlist examples

Finally, here's an example of how a repear_playlists.ini file could look like:

[Heavy Metal]
include = /Music/Metal

[Random Soft Songs]
include = /Music
exclude = /Music/Metal
exclude = /Music/Rock
shuffle = 1
sort = artist, album, title

[Audiobooks]
include = *.book.mp3

[Hot New Stuff]
new = 1
changed = 1

[Randomized]
include = /Music
shuffle = 1
sort = start count

Each playlist is specified by a header, containing the playlist name in square brackets, and a number of statements in a key = value form. In this example, we define three playlists: »Heavy Metal«, »Random Soft Songs« and »Audiobooks«.

The »Heavy Metal« playlist is a very simple one: It consists only of one include statement that tells rePear to put everything in the /Music/Metal directory (and its subdirectories) into that playlist. You could add more include statements if you like.

The »Random Soft Songs« playlist basically contains the whole /Music folder, ordered by Artist/Album/Title, but the two exclude statements remove the Metal and Rock subdirectories from that selection. So, what you get would be a collection of all your songs except Metal and Rock ones (given that you organize your music that way, of course). The shuffle = 1 statement makes rePear shuffle all the songs in that playlist. If this statement is not present, rePear would add the songs in alphabetic order, subdirectories first – just like you see them in your file manager if »sort by filename« is selected.

The »Audiobooks« playlist makes use of a generic filename pattern instead of a directory name. It selects all files whose names end in .book.mp3, regardless of how scattered they are across the iPod's directory structure, and puts them into a common playlist.

The »Hot New Stuff» playlist is a special one: The new and changed statements act like special include statements, except that they don't match filenames, but the »freshness« of a file. If the new statement is active, rePear will include every file in the playlist that has not been there when the database was last frozen (note that this will include moved files as well!). Likewise, the changed statements instructs rePear to include every file that has been changed, but not added or renamed. In combination, these two statements build a playlist that mirrors every change since the last freeze operation.

Finally, there's the »Randomized» playlist, which simply contains the whole /Music folder in random order – or almost random order, because it's then sorted by start count. This has the effect that tracks that have been played less frequently occur early in the playlist, while tracks that are played often will be put at the end of the list.

Details on artwork

rePear's artwork support tries to make sensible assumptions on what image to display for every title. Consider, for example, a track like /MyMusic/TheAlbum/TheTrack.mp3. The image (JPEG or PNG format) shown for this track will be the first match from this list (sorted from highest to lowest priority):

1. an image file with the same name as the music file, but with the filename extension .jpg or .png, like /MyMusic/TheAlbum/TheTrack.jpg
2. an image file with the same name as the directory of the music file (except the extension, of course), like /MyMusic/TheAlbum/TheAlbum.jpg
3. an image file in the same directory as the music file that contains the word »front« somewhere in its name, like /MyMusic/TheAlbum/front.jpg or /MyMusic/TheAlbum/Image_Front.jpg (this rule is there to ensure that »front« covers are prioritzed over »back« covers, even though the latter ones come first in the alphabet)
4. an image file in the same directory that contains the word »cover«, like /MyMusic/TheAlbum/cover_image.jpg
5. the first image file (in alphabetic order) in the directory, like /MyMusic/TheAlbum/SomeImage.jpg
6. an image file with the same name as the directory, but located in the parent directory, like /MyMusic/TheAlbum.jpg
7. the directoy-specific rules are inherited to subdirectories, i.e. if there is any directory-wide image file for one of the parent directories, it will be used for subdirectories, too; consider, for example, /MyMusic/MyMusic.jpg

Please read rule #7 carefully: If you have some totally music-unrelated image files in the root directory, the first of these images will be assigned to every track of the iPod (except those that have higher-priority images, of course). You probably don't want that, so keep your folders clean :)

Details on last.fm scrobbling

Some options regarding the last.fm scrobble feature can be set up in another .ini file in the iPod's root directory: repear_scrobble.ini is located right next to repear_playlists.ini and is syntactically similar. It also looks like a Windows INI file, but it lacks the sections.

The most important things to set up in this file are username and password – these are your last.fm credentials. The password can be specified either in plaintext or as a MD5 sum. Neither method is really secure, but the MD5 sum at least makes it impossible to guess the plaintext password, so this is the recommended method. Normally, you don't need to set these values anyway, as the setup options in rePear and the GUI launcher already take care of this.

However, there are additional options in this file that need to be put there by hand if they're needed: The exclude options can specify directories or filename patterns for which scrobbling shall not take place. This is useful if you don't want some of your tracks appear in your last.fm profile, like audiobooks.

A more detailed look at rePear's options and actions

As stated above, rePear automatically chooses between the freeze and unfreeze actions if it is started without parameters. However, you can override this by specifying the action as a command-line parameter (like in the dissect example: repear dissect). The following actions are available:

help

Shows the brief help message and exits the program after that.

auto

Automatically choose between freeze and unfreeze, based on the current state of the cache. This is the default if no action is specified.

freeze

Scans the iPod for playable music files, moves them into the /iPod_Control/Music directory and generates an iTunesDB from it.

unfreeze

Moves music files that have previously been moved into /iPod_Control/Music by the freeze action back to their original locations.

update

Rebuilds iTunesDB based on rePear's internal cache with the data from the last freeze. In principle, this is identical to the freeze action, except that it doesn't search for new files. However, it will update play counts, scrobble tracks to last.fm and rebuild the automatic playlists specified in the repear_playlists.ini file.

dissect

Parses the current iTunesDB and moves all tracks found there into a directory following a /Dissected Tracks/<artist>/<album>/<title> scheme.

reset

Deletes rePear's metadata cache. All information about the tracks installed on the iPod will be erased, but the music files themselves will remain. Note that if this is done while the database is in the frozen state, the information about the original filenames will be lost, too, so the files will be »trapped« in /iPod_Control/Music.

cfg-fwid

Tries to determine the serial number (FWID) of the currently attached iPod. This will also be done automatically during the first freeze operaton, but the auto-detection can also be started manually with this option.
The serial number needs to be known because some newer iPods (nano 3G, classic 6G and newer) require a checksum in the iTunesDB file that depends on the serial number.

cfg-model

Runs the (text-based) model selection wizard.

cfg-scrobble

Runs the (text-based) scrobble configuration wizard.

config

Runs all of the cfg options, one after another.

Command-line options

To get command-line help, run repear –h. This will also tell you about some other options that are available. These should be placed between repear and the action name:

• Use –r [some path] to tell rePear where the iPod's root directory is. This is useful if rePear can't determine from what directory it was called, or if you deliberately don't want to keep rePear in the iPod's root directory.
• –l [some filename] specifies where the rePear logfile should be written to.
• –L [options] can be used to override the LAME encoding options that are used when transcoding Ogg files.
• Usage of the option –m [model] is required for the cover artwork feature. This option is only used for the freeze action, and its value is saved for upcoming freeze actions that won't need this option again. It can also be set using the configuration wizard or the GUI. Valid models are:

  nano, nano1g or nano2g	iPod nano, first or second generation
  4g or photo	iPod photo (4th generation)
  5g or video	iPod video (5th generation)
  6g, classic or nano3g	iPod classic (6th generation) or iPod nano third generation (»fat nano«)
  nano4g	iPod nano 4th generation

• –f deactivates the confirmation prompts that are shown when doing »uncommon« things.
• –p [some filename] specifies the location of the master playlist file.
• –s [some filename] specifies the location of the scrobble configuration file.
• On Windows systems, rePear will wait for a keypress after it is done. The ––nowait option deactivates this behavior.

iPod, iTunes and Mac OS X are registered trademarks of Apple Inc. Windows is a registered trademark of Microsoft Corp.