Slacy’s Blog

I’m running my own hand-crafted remote backup (rsync++) script to back up data from one machine to another.  But, it has some serious drawbacks, and I’m still looking for something better.  Here are my requirements:

• Runs on Ubuntu
• Is either:
  • Available in the standard distro (i.e. via apt-get)
  • A simple self-contained script (Python, Perl, Bash, etc.)
• Uses ssh/scp/rsync to transport files
• Is incremental (i.e. does “what’s changed from last time”, and only backs up that much)
• Handles file renames & moves without re-transmitting the underlying data.
• Puts files in a simple “directory by date” format.  No fancy databases, no fancy metadata required for restoring.
• Preserves file permissions & modification dates.
• Correctly handles both hard & soft symbolic links.
• Is client-side only.  In other words, I don’t want to have to maintain 2 copies of this script, one for the server, and one for the client.

Is there anything out there that meets these requirements?

I was having the following issue while doing a “git push”:

$ git push
Counting objects: 50, done.
Compressing objects: 100% (33/33), done.
Writing objects: 100% (36/36), 4.42 KiB, done.
Total 36 (delta 13), reused 0 (delta 0)
error: unable to write sha1 filename ./objects/fc/0bf175cf9bf2ecdf15eee84adad32230107aa7: Permission denied

fatal: failed to write object
error: unpack failed: unpacker exited with error code
To slacy@myhost.com:/home/git/name-of-repo
 ! [remote rejected] master -> master (n/a (unpacker error))
error: failed to push some refs to 'slacy@myhost.com:/home/git/name-of-repo'

The issue turned out to be that because I’m using ssh + git to push the files, the files in the .git/objects directories are created with the user+group of the user on the host.  So, the first user to create the file makes them with their own userid & groupid, and are set with default permissions as u=rxw,g=rx,o=rx, and thus, when the second user comes it to add a new file to that pre-existing subdirectory, they don’t have permissions.

I tried to solve this issue by creating “group git” and adding all users who are using the repository in /home/git/name-of-repo, but it still turns out that new files are created with userid:groupid as user:user not user:git.

How do you get around this issue?  How do I make it so that new directories created by git during the push process are given the correct permissions (u=rxw,g=rxw,o=) and assigned to group git instead of the user group?

This issue is easy enough to fix manually, by doing something along the lines of:

$ ssh userid@myhost.com
(myhost.com)$ cd /home/git/name-of-repo/.git/objects
(myhost.com)$ find . type d | xargs chgrp git
(myhost.com)$ find . type d | xargs chmod g=rxw

But, I’m going to have to continue to do this as new directories in the “objects” area are going to continue to be created with the wrong permissions.

GIT HAS THE WORST ERROR MESSAGES IN ANY PIECE OF USER-FACING SOFTWARE I’VE EVER SEEN.

There, I said it. Peace out.

UPDATE:

Just read about the git config setting “core.sharedRepository” which will automatically give group rwx permissions to newly created files in the repository.  So, I ran:

$ git config --add core.sharedRepository group

When I stream music from my home to my work, I want to make sure that I throttle my music such that it doesn’t saturate my outgoing network connection and make the rest of my internet slow.

All I wanted to do was throttle a stdout pipe that’s being sent to the music player.  I found reference to a utility called throttle, but also found that it isn’t included in the Ubuntu 9.04 distribution.

There’s a maliing list post that says that the program cstream will do the same thing.

man cstream says:

-t num    Limit the throughput of the data stream to num bytes/second. Limiting is done at the input side, you can rely on cstream not accepting more than this rate. If the number you give is positive, cstream accumulates errors and tries to keep the overall rate at the specified value, for the whole session. If you give a negative number, it is an upper limit for each read/write system call pair. In other words: the negative number will never exceed that limit, the positive number will exceed it to make good for previous underutilization.

So, I changed my audio transcoding script to say:

lame –quiet -V6 $FILE – | cstream -t 25000

The number “25000″ comes from the fact that I don’t want to use more than 192kbps of my outbound stream, and cstream’s argument is in bytes per second.  192kbps = 24000 bytes/second, and I added a little extra.  It would probably be safer if I also passed “-B 192″ to lame, but that would actually limit my overall quality, so I’m just going to cross my fingers and hope that “-V6″ doesn’t produce >192kbps for very long.

Introduction

mt-daapd is an iTunes daap-protocol server for Unix systems.  I’m using mt-daapd on by Ubuntu 9.04 system to stream music over the internet to my workplace.   The issue is that many of my mp3 files are 256kbps mp3 files, and streaming these takes up more bandwidth than necessary.

mt-daapd includes a transcoding system called “ssc” for “Server Side Conversion”.  This system works by either a shared-object plugin (i.e. ssc_ffmpeg) or via an external script file (i.e. ssc_script).

using the ssc_script functionality, one can invoke ‘lame’ (the mp3 encoder) to transcode high bitrate mp3 files into lower bitrate mp3 files.  The setup for this in mt-daapd wasn’t very obvious, so I’ll document it here.

Install mt-daapd

The version of mt-daapd provided by Ubuntu 9.04 includes support for transcoding, so you can just run:

$ sudo apt-get install mt-daapd

Turn on ssc

Edit your /etc/mt-daapd.conf file, and make sure you have the following lines in the correct places:

ssc_codectypes = mpeg

always_transcode = mpeg

ssc_prog = /path/to/script/shown/below/mt-daapd-mp3-ssc.sh

[plugins]

plugins = ssc-script.so

Create mt-daapd-mp3-ssc.sh

You’ll need to create a shell script to do the transcoding via ssc_scrpit / ssc_prog.  Here’s the script I’m using:

#!/bin/bash

FILE=$1
OFFSET=0

if [ "$2" == "" ]; then
OFFSET=0
else
OFFSET=$2
fi

if [ "$3" == "" ]; then
FORGELEN=$3
fi

lame -V6 –quiet “$FILE” -

Note that I’m looking at the 2nd and 3rd args, but I’m not doing anything with them.  These arguments are used to make sure that seeking works properly in your daap client.  I don’t care about seeking, and doing it properly is somewhat hard, so I’ve just ignored those arguments.  I think that this could be accomplished properly using the program ‘mp3splt’ but I haven’t looked into it enough to see what it would take.

Make sure the script above is put somewhere accessible by the user specified as ‘runas’ in your mt-daapd.conf.   Make sure the script is executable by this user as well.

Restart mt-daapd and see if it works

Restart mt-daapd via:

$ sudo /etc/init.d/mt-daapd restart

Then reconnect your daap client (I use rhythmbox).  You can see if the script is working by running “ps auwxww | grep lame” just after pressing play on a song.  You should see your script executing lame and doing realtime transcoding.

It was a long and tumultuous affair, but now I’ve officially given up on web-based streaming of my personal music collection.

I started out with netjuke, which died and got absorbed into the horrible jinzora project.   So then I switched to Ampache, which worked fairly well, but needs some serious UI upgrades/changes to make it reasonably useful.

Now that I’m fully ubuntu-ized, I’m using mt-daapd through an SSH tunnel to listen via a native client app.  This solution has lots of interesting benefits:

1. Running mt-daapd locally means other people in my house can see my music, including my TiVo and any other desktops with an iTunes/daap compatible frontened.
2. I don’t have to worry about security issues with having all my music accessible via a web portal that could be hacked.
3. mt-daapd is available as a package in Ubuntu, so that means upgrades are easy.  Doing upgrades (by hand) of php-based web apps was really becoming a drag.
4. This means less junk on my slacy.com web server, which I’m trying to significantly trim down and make more secure by having less applications.

As far as I can tell, there are no RRDTool bindings for php5 for Ubuntu 9.04.  It appears as though it existed in prior distributions that included php4, but Ubuntu 9.04 no longer includes php4.

Several others are talking about compiling it from source, but this seems very wrong and unnecessary to me.

If you have any other information about how to easily via apt-get make RRDTool functions work in php in Ubuntu 9.04, please leave a comment here.

I’m not sure why this has to be so complicated.

Just some quick notes before I forget them.

I’ve recently switched my primary desktop machine from Fedora Core 8 (very old, I know) x86_64 to Ubuntu 9.04 (latest and greatest) x86_64.  In general, things went well, but there were a couple gotchas that I had to work through, so I thought I’d list them here:

• The default Ubuntu 9.04 desktop installer won’t let you set up RAID by default.  So, you have to boot and run the “alternate” installer.
• If you use unetbootin to “burn” the Ubuntu alternate install disk to a USB stick and boot from that, then it will have errors during the install asking you to “insert your CD-ROM”.  Ugh.  The solution here is to select “Install” from the unetbootin boot menu instead of “default”.  This will run a network-based install instead of trying to install from the USB image that you just burned.  I know it’s silly, but at least it worked.
• Ubuntu, unlike Fedora, only allows x86_64 for it’s libraries in /lib, unlike Fedora which puts 32-bit libraries in /lib and 64-bit ones in /lib64.  Ubuntu has an option (ia32-apt-get and ia32-libs-tools) to put some 32-bit libraries in /lib32, but I found that this led to some serious apt-get dependency issues, and I quickly gave up and uninstalled these packages (and removed the damage they caused).
• There are a couple of critical things that are 32-bit only:
  • The amazon mp3 downloader application.  There are some tutorials around about how to use a program called “getlibs” to make it work right, and there are some pkgbuild and wacky .deb creation scripts to make this seem to work.  It’s my understanding that both methods pollute /lib with 32-bit libraries, so I haven’t tried it yet.  I have an e-mail out to the amazonmp3 downloader maintainer, and we’ll see if that pans out.
  • Adobe Flash.  Thankfully, Ubuntu seems to have worked this one out, and if you “sudo apt-get install flashplugin-installer” it just works, although it uses ndispluginwrapper, which is sort of a giant hack, but at least it works until Adobe gives full support for 64-bit flash on Ubuntu.
• Although the Ubuntu 9.04 fonts looked pretty good out of the box, I found that my installing some alternate font packages, that I could really improve the look of many web pages.  The packages that I installed were: msttcorefonts, ttf-bitstream-vera, ttf-dejavu-core, ttf-droid, ttf-liberation, ttf-xfree86-nonfree.  I think that’s a fairly good list, and was surprised that they weren’t installed by default.

So, that about sums it up so far.  I’m running ext4, on an md RAID5 array, and things seem to be reasonably stable thus far.

Please note that I did not install Ubuntu on top of Fedora.  I’m not sure what would happen if one were to do this.  I did a clean install on fresh disks, and then copied my data back from a backup.

I listen to all of my music via streaming it from my home machine, and transcoding it on the fly. I recently made 2 changes to my setup:

1. I’m using audacious for playback instead of xmms
2. I switched to a more recent version of lame.

After doing so, about 1 in 10 files would fail to play (either silence, or strange thumping noises) on audacious, saying one or more of the following messages:

(audacious:12805): MADPlug-WARNING **: samplerate varies!!
(audacious:12805): MADPlug-WARNING **: layer varies!!
(audacious:8645): MADPlug-WARNING **: number of channels varies!!

If I downloaded the file and played that, it worked just fine, so it was something weird about the fact that it was streaming.  My lame transcoding commandline was:

lame –mp3input -h –vbr-new -V 6 -B 320 -b 32 -S -m j $file -

The problem seems to be –vbr-new, since if I remove that option, it works great all the time.  Go figure.

In general, I think audacious sucks.  Now I also think that madplug sucks.  Oh, and by the way, all of this was precipitated by Fedora removing xmms from their repositories due to mp3 licensing issues.  Fedora sucks too.  Ugh!  It seems as though mp3 playback under Linux has taken a huge step backwards…

Some collected links.  I have yet to get this working properly.  The osx86 install images hang at “dsmos: Starting”

http://d4wiki.goddamm.it/index.php/Howto:_Mac_OSX_on_KVM

It seems as though KVM/qemu itself needs to be patched to emulate the AppleSMC hardware device.  I can’t yet find out of the current Ubuntu 7.x or 8.x distros of KVM include this patch or not.  It does seem like this has already made it to opensuse, and the patch is nearly a year old, so maybe it’s in Ubuntu 8.x or 9.x:  Here’s the patch itself:

http://svn.exactcode.de/t2/trunk/package/emulators/kvm/04-qemu-applesmc.patch

Unfortunately, the author’s website (http://alex.csgraf.de/self/) seems to be really borked and no longer has the relevant information other than the “getsmc” script referenced from the wiki page.  (Not sure if this getsmc script is for Apple h/w + OSX or Apple + Linux)

There is an “applesmc” kernel module for Ubuntu, but I suspect this is for accessing the *real* SMC device, not for emulating one on non-Apple hardware.

So, the Lenovo S10 works great with Linux, particularly Ubuntu 8.10, but the one thing thats a bit wonky is the way the trackpad works.  The trackpad is non-square, but reports square coodinates, so the vertical motion is exaggerated.

I’ve compiled a hacked version of the X11 Synaptics driver that includes a fudge factor for the Y component.  It does hw.y *= 0.6 internally, and this pretty much compensates for the incorrect coordinates.

Here’s a link to a modified replacement Ubuntu .deb package.

Just download that file, and then run:

dpkg –install xserver-xorg-input-synaptics_0.15.2-0ubuntu7_i386.deb

And you should be good to go.  If/when the Ubuntu maintainers publish a new or upgraded version of this package, this version will be overwritten, so you’ll be SOL.  So, be careful with those autoupgrades!

In other news, there are patches underway to do this in a more configurable way, and its possible that future versions of the Synaptics driver will include the ability to adjust the vertical sensitivity.