Using cookies.sqlite in wget or curl

Newer versions of FireFox use a sqlite3 database to store their cookies.

Older versions of FireFox used a .txt file.  wget and curl know how to parse the older style text file, but not the sqlite3 database.

So, I wrote a quick script to extract the cookies.sqlite database and generate a file that looks just like the old cookies.txt file, and then you can pass that to wget or curl. Here’s the script:

#!/bin/bash

function cleanup {
rm -f $TMPFILE
exit 1
}

trap cleanup  SIGHUP SIGINT SIGTERM

# This is the format of the sqlite database:
# CREATE TABLE moz_cookies (id INTEGER PRIMARY KEY, name TEXT, value TEXT, host TEXT, path TEXT,expiry INTEGER, lastAccessed INTEGER, isSecure INTEGER, isHttpOnly INTEGER);

# We have to copy cookies.sqlite, because FireFox has a lock on it 
TMPFILE=`mktemp /tmp/cookies.sqlite.XXXXXXXXXX`
cat $1 >> $TMPFILE
sqlite3 -separator '	' $TMPFILE << EOF
.mode tabs
.header off
select host,
case substr(host,1,1)='.' when 0 then 'FALSE' else 'TRUE' end,
path,
case isSecure when 0 then 'FALSE' else 'TRUE' end,
expiry,
name,
value
from moz_cookies;
EOF
cleanup

Usage should be like this:

$ extract_cookies.sh $HOME/.mozilla/firefox/*/cookies.sqlite > /tmp/cookies.txt
$ wget --load-cookies=/tmp/cookies.txt http://mysite.com
$ # OR 
$ curl --cookie /tmp/cookies.txt http://mysite.com 

You could use this script to wrap curl or wget itself in a similar extraction scheme, so that you always get the most current cookie values when you run wget.

12 thoughts on “Using cookies.sqlite in wget or curl”

  1. Thanks. One minor problem, if there’s a space in your cookie jar path, the script fails.

    To fix it, change:

    cat $1 >> $TMPFILE

    to

    cat “$1″ >> $TMPFILE

  2. Awesome plugin, I use it almost every day. Thank you so much, Zaid!
    I didn’t know that you have to type ‘about:config’ into the URL bar to get to the firefox configuration attributes hope this information helps others as well.

  3. Hey. Awesome script. This does feel like the sort of thing that should get included with firefox, but ah well.

    I’ve hacked my version of the script so that it looks up the path to a given profile like so:

    profile_name=$1
    path_to_profile=$(cat $MOZ_PATH/profiles.ini | grep -E “^Name=|^Path=” | grep “^Name=${profile_name}\$” -A 1 | grep Path= | cut -d ‘=’ -f 2-)

    Also you can use the “fish operator” so that I don’t have to mess around with the intermediate cookie file

    curl -b <(extract_cookies.sh profilename)

    and I have an alias firecurl which adds these arguments by default…

  4. Hmm. This approach also doesn’t seem to work for session cookies (cookies with an empty expiry date that will be cleared when you leave a browser). These cookies are often used for authentication.

    Firefox appears to keep these in a json file called sessionstore.js, which is cleared when firefox exits. (Splitting cookie stores in two strikes me as a terrible idea.. – but at least you can get at these cookies).

    The following python script will extract the cookies from a session store.

    #!/usr/bin/env python
    import json
    import sys
    session = json.load(sys.stdin)

    cookies = set()

    for window in session["windows"]:
    if ‘cookies’ not in window:
    continue
    for cookie in window["cookies"]:
    cookies.add(tuple(cookie.items()))

    for cookie in cookies:
    cookie = dict(cookie)
    print “\t”.join([cookie["host"], str(cookie["host"][0] == “.”).upper(), cookie["path"], “FALSE”, “0″, cookie["name"], cookie["value"]])

    You can then use this as part of the extract cookies script

    I still had problems for my use case though, as the website helpful detected that something was different between requests (user agent maybe?), and logged me out :(.

Leave a Reply