Read FireFox Cache with Python

Do you know the easiest way to view the disk cache data of your firefox browser? Just paste "about:cache?device=disk" (without the quotes) in the address bar of your firefox browser. If you see it for the first time, you will be amazed.

Anyway, I had to write a program that does similar thing (locate and read firefox cache and show cache data) in Python. First I tried to automate the about:cache mechanism, but couldn't make it. Later I went for processing raw cache data.

In the Cache directory, the files _CACHE_001_, _CACHE_002_, and _CACHE_003_ contain a mix of data and metadata. The metadata contains information such as the URL requested, timestamps, size, and other details. You can view the files (I used VI in my Ubuntu to view those).

The file structure of each _CACHE_00n file is as follows: 

 ______________________________
|                              |
|       4096 byte free         |
|        block bit-map         |
|------------------------------|
|                              |
|            Blocks            |
|             ...              |
|                              |
|                              |
|______________________________|
The size of each block varies depending on the file. 
_CACHE_001_          256 byte blocks
_CACHE_002_         1024 byte blocks
_CACHE_003_         4096 byte blocks

Data in the cache is always encoded in a group of 1-4 blocks and always starts exactly on a block boundary (an exact multiple of 256, 1024, or 4096 bytes). For cache data that corresponds to metadata, it is encoded in this very specific data format. 

      _________________
0-3  |       magic      |    unsigned int   (0x00010008)   (Big Endian)
     |------------------| 
4-7  |    location      |    unsigned int  (Big Endian)
     |------------------|
8-11 |    fetchcount    |    unsigned int  (Big Endian)
     |------------------|
12-15|    fetchtime     |    unsigned int  (Big Endian)
     |------------------|
16-19|    modifytime    |    unsigned int  (Big Endian)
     |------------------|
20-23|    expiretime    |    unsigned int  (Big Endian)
     |------------------|
24-27|    datasize      |    unsigned int  (Big Endian)
     |------------------|
28-31|    requestsize   |    unsigned int  (Big Endian)
     |------------------|
32-35|    infosize      |    unsigned int  (Big Endian)
     |------------------|
36-  |  request string  |    char[].  Length is determined by
     |                  |    the value of the requestsize field
     |       ...        |    above.  Terminated by a \x00 byte.
     |                  |
     |                  |
     |------------------|
n-   |   info string    |    char[].  Length is determined by
     |                  |    the value of the infosize field
     |       ...        |    above. 
     |                  |
     |                  |
     |                  |
     |__________________|

But interestingly I didn't need to use all these knowledge. Just used regular expression to parse the data :-)

Comments

Patrick said…
Thanks for the post. This is useful as I am trying to do the same thing in python. Is there any code that you can share by any chance on this?
April 19, 2008 at 10:58 PM
Soroush said…
thx for ur nice blog content
can u explain file structure of _CACHE_MAP_ file as well plz ?
It`s now critical for me :)
November 8, 2008 at 1:51 PM
Vinod said…
Thanks
I am using Asp.net in my project,can you tell me how should i open the data in those file.
is data contains any images?

please reply
August 30, 2011 at 3:29 PM
Unknown said…
lsamI have been searching for this for past 1 month.. Me too have to do the same for my Project..
How to implement this using Python.. Can u share the code.?

Or can u xplain how to read/parse data from index.bat file or How IExplore works..

Expecting ur help by anycase..
February 25, 2012 at 6:35 PM
Post a Comment

Popular posts from this blog

Strip HTML tags using Python

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags: def remove_html_tags(data): p = re.compile(r'<.*?>') return p.sub('', data) Here is another function to remove more than one consecutive white spaces: def remove_extra_spaces(data): p = re.compile(r'\s+') return p.sub(' ', data) Note that re module needs to be imported in order to use regular expression. Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more

Convert text to ASCII and ASCII to text - Python code

Python has a very simple way to convert text to ASCII and ASCII to text. To find the ASCII value of a character use the ord() function. Example: >>> ord('B') 66 To get a character from it's ASCII value use the chr() function. Example: >>> chr(65) 'A' Here is a program to get the list of all characters along with their values: # program to print the list of characters and their ASCII values for value in range(0, 255): print "ASCII Value:", value, "\t", "Character:", chr(value), "\n" Here is another simple ASCII encoding-decoding program: # Convertion from text to ASCII codes message = raw_input("Enter message to encode: ") print "Decoded string (in ASCII):" for ch in message: print ord(ch), print "\n\n" # Convertion from ASCII codes to text message = raw_input("Enter ASCII codes: ") decodedMessage = "" for item in message.split(): decodedMessage += chr
Read more