Breaking Code

September 19, 2012

Quickpost: Cheating on XKCD

Filed under: Just for fun — Tags: , , — Mario Vilas @ 3:23 pm

In case you missed it, today’s XKCD comic titled Click and Drag is simply amazing! Go check it out first, spend a few hours lost in it, and come back only when you’re done having fun. I’ll wait here. :)

Ok, you’re back. Naturally you’ll want to cheat on it at some point, to make sure you didn’t miss out on any hidden easter eggs! So let’s take a look at the web page.

The easiest route is loading the comic on Google Chrome, or Chromium. Just right click on the image and select “inspect element”. This quickly reveals how the neat trick works.

Taking a peek under the hood...

Taking a peek under the hood…

The “world” is divided into tiles of fixed size, and at all times the page loads the tile you’re currently viewing and the surrounding ones, in order to seamlessly stitch them together when scrolling. The clickable area is a map and the coordinates are used to build the URL to the images, which always follows the same pattern (north, south, and east and west coordinates). Trying out a few numbers reveals the “north” coordinate goes from 1 to 5, the “east” coordinate goes from 1 to 48 and the “west” coordinate goes from 1 to 33. Not all coordinates seem to work around the edges of the world (north 2 west 5 doesn’t work for example) and I couldn’t get south to work with manual tries. I suppose a couple empty images are used for those (one for black and one for white) but I didn’t confirm it.

The first thing I tried was just accesing the parent directory to see if directory indexing was enabled, but no such luck. Instead, I wrote this quick and dirty script in Python to download all images, using urllib to download them and shutil to write them to disk. Missing tiles are simply skipped.

This should be enough to check for easter eggs, but it’d be interesting of someone assembles a big image containing all the tiles. Let me know if you do! :)

Update 1: I originally missed the east coordinate, so the script was updated to try and bruteforce in all directions 1 to 10 north and south, and 1 to 50 east and west. This means a lot more HTTP requests, so I also added a pause between them as good netizens should.

Update 2: This seems to be the complete list of valid image URLs.

Update 3: A commenter pointed out somebody did assemble the entire world image! Check it out here.

Update 4: @prigazzi on Twitter pointed out this fully navegable map as well, based on Google Maps. Check it out! It’s IMHO the best one yet. :)

Update 5: The previous link no longer works, but this works pretty much the same way: xkcd-map.rent-a-geek.de

(more…)

January 27, 2012

Posting anonymously to Pastebin.com

Filed under: Privacy, Tools, Web applications — Tags: , , , , , , , , , — Mario Vilas @ 6:58 pm

Updated 3-Jun-2013: the API this script was using has been deprecated, and the new one requires credentials to be used. :(

I’m keeping the blog post online for a while as a curiosity, but the technique described here no longer works.


While going through some old code of mine to document it using Epydoc at a friend’s request, I found something funny. Some time ago I made a quick and dirty script to access the Pastebin.com API from Python. Well, it turns out the API has changed quite a bit since then – most importantly, now it requires a mandatory API key that’s linked to a user account (which in turn, if you used OAuth, is linked to your Gmail address or Twitter feed). That means it’s no longer possible to post anonymously using the official API.

Funny thing is, my old script was still working! :) Apparently the folks at Pastebin have left the legacy API still running. The old documentation is gone though, and now even to read the updated documentation you need to log in… :(

Now, this takes care of the API key problem, but there’s still the issue of Pastebin seeing your IP address. An HTTP proxy can fix that… provided you trust that proxy not to store your IP somewhere in the logs. The procedure is simple, just set the HTTP_PROXY environment variable to wherever your proxy is, and voilà! The standard Python module urllib will automatically connect through the proxy.

If you don’t have a trusty HTTP proxy, the best way to go is through the Tor network. You’ll need to install the Tor service itself and the Privoxy HTTP proxy in your machine, then set the HTTP_PROXY variable to 127.0.0.1:8123. This document from the Tor Project explains it in detail.

Once you’ve set up your proxy, download pastebin.py and send a file to Pastebin like this:

    $ python pastebin.py manifesto.txt
    manifesto.txt --> http://pastebin.com/ixSetT5f

The script accepts multiple filenames as well. You can also set the syntax highlighting format (useful for source code, config files or logs) as follows:

    python pastebin.py --format=apache /var/log/apache2/access.log /var/log/apache2/error.log

And set an expiration time, after which it gets automatically deleted. In the following example a SQL dump is uploaded and automatically deleted the next day:

    python pastebin.py dump.sql --format=sql --expire=1D

So, that’s pretty much it. There’s a limit of 512Kb per file uploaded, in order to bypass this you’ll have to split the file into multiple pieces (something similar to this but using pastes instead of URLs). I may do it another day, but for now it’s left as an exercise to the reader. ;)

There’s one thing I don’t quite understand: why did the Pastebin folks think it was necessary to have a mandatory API key? Even if the legacy API had been shut down, it would still be possible to figure out how the web page was doing it and replicate it in Python. The API key being linked to the user account seems a bit strange too… Their intention might be to catch script kiddies uploading illegal stuff, but it may also be an attempt to do data mining on people’s posts. Who knows…

Download


pastebin.py

No longer works!

June 29, 2010

Quickpost: Using Google Search from your Python code

Filed under: Tools, Web applications — Tags: , , , , , , , , — Mario Vilas @ 6:31 pm

Hi everyone. Today I’ll be showing you a quick script I wrote to make Google searches from Python. There are previous projects doing the same thing -actually, doing it better-, namely Googolplex by Sebastian Wain and xgoogle by Peteris Krumins, but unfortunately they’re no longer working. Maybe the lack of complexity of this script will keep it working a little longer… :)

The interface is extremely simple, the module exports only one function called search().

        # Get the first 20 hits for: "Breaking Code" WordPress blog
        from google import search
        for url in search('"Breaking Code" WordPress blog', stop=20):
            print(url)

You can control which one of the Google Search pages to use, which language to search in, how many results per page, which page to start searching from and when to stop, and how long to wait between queries – however the only mandatory argument is the query string, everything else has a default value.

        # Get the first 20 hits for "Mariposa botnet" in Google Spain
        from google import search
        for url in search('Mariposa botnet', tld='es', lang='es', stop=20):
            print(url)

A word of caution, though: if you wait too little between requests or make too many of them, Google may block your IP for a while. This is especially annoying when you’re behind a corporate proxy – I won’t be made responsible when your coworkers suddenly develop an urge to kill you! :D

Below are the download links (source code and Windows installers) and the source code for you to read online. Enjoy! :)

Changelog

  • Version 1.0 (initial release).
  • Version 1.01 (fixed the IOError exception bug).
  • Version 1.02 (fixed the missing href bug reported by Rahul Sasi and the duplicate results bug reported by Slawek).
  • Version 1.03 (extracts the hidden links from the results page, thanks ubershmekel!).
  • Version 1.04 (added support for BeautifulSoup 4, thanks alxndr!).
  • Version 1.05 (added compatibility with Python 3.x, better command line parser, and also added some improvements by machalekj)
  • Version 1.06 (added an option to only grab the relevant results, instead of all possible links from each result page, as requested by user Nicky and others).

Download

Source code: google-1.06.tar.gz

Windows 32 bits installer: google-1.06.win32.msi

Windows 64 bits installer: google-1.06.win-amd64.msi

Documentation: google-1.06-doc.zip

Source code

Get the source code from GitHub: https://github.com/MarioVilas/google
(more…)

March 6, 2010

Quickpost: Using the Pastebin.com API with Python

Filed under: Tools, Web applications — Tags: , , , , , — Mario Vilas @ 11:59 pm

Hello everyone. Today we have a quick script I just coded to access the Pastebin.com API from Python. It can be imported like a module or run as a command line script, and the interface is really simple (just one function!). The command line interface allows you to upload files. Enjoy! :)

Edit: Also see this post on how to use this script to send files anonymously to Pastebin.

Updated 28-Jun-2011: small changes made
Updated 27-Jan-2012: fixed bug in paste_expire_date, added Epydoc documentation
Updated 28-Jan-2012: removed paste_subdomain, feature seems to have been deprecated
Updated 3-Jun-2013: the API this script was using has been deprecated, and the new one requires credentials to be used. :(

Download


pastebin.py

The API this script was using has been deprecated, and the new one requires credentials to be used. :(

Source code

(more…)

January 11, 2010

Having fun with URL shorteners

Filed under: Tools, Web applications — Tags: , , , , — Mario Vilas @ 2:14 am

I’ve taken some interest in URL shorteners recently. URL shorteners are the latest fad in web services – everybody wants to have their own, even if it’s not yet entirely clear how to profit from it, at least not for us mere mortals. They were born out of Twitter users need to compress their microblogging posts (some links are actually longer than 160 characters, believe it or not) and it’s spread everywhere on the Internet. Major web sites like Facebook, Google and Youtube are into it and -some say- it’s one step closer to destroying the Web as we know it.

Of course, this can also be used for evil purposes. :)

So I made myself a Python module to toy with these things. It can be used as a command line tool as well as a module to be imported in projects of your own, and contains no non-standard dependencies. You can download it from here: shorturl.py.

For example, this is how you convert a long URL into a short one (the shortest possible choice is automatically selected):

    $ ./shorturl.py http://www.example.com/
    http://u.nu/63e

And it’s reverse, converting that short URL back into the long one:

    $ ./shorturl.py -l http://u.nu/63e
    http://www.example.com/

You can also choose a specific URL shortener service. Let’s try Twitter’s new favorite, bit.ly:

    $ ./shorturl.py http://www.example.com/ -u bit.ly
    http://bit.ly/3hDSUb

Now, the fun stuff :) what would happen if we try to shorten, say, a javascript: link? We can find out with the -t switch (output edited for brevity):

    $ python shorturl.py -t "javascript:alert('Pwn3d');"
    Testing bit.ly:
    RuntimeError: No data returned by URL shortener API
    Testing cru.ms:
    RuntimeError: <h3>URL has wrong format! Forgot 'http://'?</h3>
    Testing easyuri.com:
            Short [1]: http://easyuri.com/657ae
            Long  [0]: http://javascript:alert('Pwn3d');
    Testing tinyurl.com:
            Short [1]: http://tinyurl.com/yhabhth
            Long  [0]: javascript:alert('Pwn3d');
    Testing xrl.us:
            Short [1]: http://xrl.us/bgsge5
            Long  [0]: http://xrl.usjavascript:alert('Pwn3d');

Most services either report an error condition or simply don’t return anything at all. Two services return a garbage response (easyuri.com and xrl.us). But to my surprise TinyURL, the second largest URL shortener service is vulnerable to this! True, this is probably not news to many of you, and blocking this kind of links doesn’t provide that much of an advantage (clicking on a link with an unknown target is pretty much game over anyway) but I still fail to see the reason to support it in the first place. I can’t think of any legitimate reasons to shorten a javascript link.

Example of TinyURL's poor handling of javascript links

As a matter of fact, TinyURL seems to accept anything you send it, may it look like a valid URL or not. This has lead some people to build a rogue filesystem called TinyDisk, world readable, yet very hard (if not impossible) to trace back to it’s creator. The download link seems to be down but the Slashdot article is still there. I also found this article explaining how such a filesystem would work.

Back on the matter, the way people react to URL shorteners make them perfect for an attacker to hide the true target of his/her exploits, or bypass spam filters as it’s being done in the wild for quite some time now. There are some countermeasures. Many URL shorteners (including TinyURL) have a preview feature to let you examine the URL target before going there (for example, http://preview.tinyurl.com/yhabhth shows the sample javascript code we used above). The Echofon plugin for Firefox is a Twitter client that makes heavy use of URL shorteners, and among other things it automatically expands short URLs when the mouse hovers over them. The longurl.com service offers URL expansion as well, both from the web and as a Firefox plugin.

But… how well do they behave when you shorten the URL multiple times? First, let’s try shortening a link with TinyURL three successive times:

    $ ./shorturl.py -c 3 -u tinyurl.com http://www.example.com/
    http://tinyurl.com/ya3k2yy

The preview feature only seems to work for the first redirection (see for yourself). Now let’s try shortening our example URL three times, using random providers:

    $ ./shorturl.py http://www.example.com -v -c 3
    Service: thurly.net
    Service: ito.mx
    Service: cru.ms
    http://cru.ms/53c14

I found the Echofon plugin fails to expand this short URL. However the longurl.com service works like a charm – it even told me how many redirections I used! Click here to see it for yourself.

Example of Echofon expanding a short URL

Just for fun, how about an infinite recursion? First, we create a link to a shortened URL that doesn’t yet exist.

    $ python shorturl.py -v -c 5 http://tinyurl.com/thisisaninfiniteloop
    Service: ito.mx
    Service: migre.me
    Service: thurly.net
    Service: is.gd
    Service: xrl.us
    Service: onodot.com
    http://onodot.com/xeaw

Now let’s go to the TinyURL webpage and create the link manually. Voila! We have an infinite redirection. The longurl.com service seems to break now (see for yourself) but this is of no consequence – the link doesn’t really take you anywhere. Other apps like wget get stuck in an infinite loop however. This is probably of no real use to an attacker but I thought it was fun to try it out. :)

As a final note: while writing this I came across this article describing even more risks inherent to URL shorteners: information disclosure. A dedicated attacker could dump randomly chosen shortened URLs to retrieve information like corporate intranet links, user IDs and even passwords. I liked the idea, so I whipped out a small script to test it out:

    from random import randint
    from shorturl import longurl, is_short_url

    characters = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
    while 1:
        token = ''.join([ characters[ randint(0, len(characters) - 1) ] for i in xrange(6) ])
        try:
            url = longurl('http://bit.ly/%s' % token)
        except Exception:
            print token
            continue
        if not is_short_url(url):
            print "%s => %s" % (token, url)

But the results of bruceforcing didn’t seem so great. Then again, one could also wonder how random this short URLs really are… maybe there’s a way to predict them, or at least reduce the number of URLs one has to try. One could also try a dictionary based approach for services like bit.ly, which provide the users the ability to pick custom short URLs. Update: Those TinyURL links weren’t so random after all ;) here’s an interesting project on URL scraping.

Read Part 2: parasitic storage

Download

shorturl.py

bitlybf.py

Source code

You can get the source code at Github.

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 2,480 other followers