jan.schnasse

Recently I introduced a solution for URL encoding in Java .

public static String encode(String url) {
    try {
      URL u = new URL(url);
      URI uri = new URI(u.getProtocol(), 
                        u.getUserInfo(), 
                        IDN.toASCII(u.getHost()), 
                        u.getPort(), 
                        u.getPath(),
                        u.getQuery(), 
                        u.getRef());
      String correctEncodedURL = uri.toASCIIString();
      return correctEncodedURL;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
}

 

Now I like to introduce a set of URLs to test the code.  Good test sets are provided at the  ‘Web Platform Tests’ (wpt) repository.  A comprehensible assembly of infos about the URL standard can be found at whatwg.org.

On basis of the ‘Web Platform Tests’ I created a file to hold test urls together with the expected outcome. The test set is provided in the following form:

{ 
  "in" : "http://你好你好.urltest.lookout.net/",
  "out" : "http://xn--6qqa088eba.urltest.lookout.net/" 
}, 
{ 
  "in" : "http://urltest.lookout.net/?q=\"asdf\"", 
  "out" : "http://urltest.lookout.net/?q=%22asdf%22" 
}

To test my URL encoding implementation I use the following code

try (InputStream in = Thread.currentThread().getContextClassLoader()
        .getResourceAsStream("url-succeding-tests.json")) {
  ObjectMapper mapper = new ObjectMapper();
  JsonNode testdata = mapper.readValue(in, JsonNode.class).at("/tests");
  for (JsonNode test : testdata) {
    String url = test.at("/in").asText();
    String expected = test.at("/out").asText();
    String encodedUrl = URLUtil.encode(url);
    org.junit.Assert.assertTrue(expected.equals(encodedUrl));
  }
} catch (Exception e) {
  throw new RuntimeException(e);
}

During my tests I also found some URLs that were not encoded correctly. I collected them in another JSON file.

Here are some failing examples:

{
  "in" : "http://www.example.com/##asdf",
  "out" : "http://www.example.com/##asdf"
},{
  "in" : "http://www.example.com/#a\nb\rc\td",
  "out" : "http://www.example.com/#abcd"
}, {
  "in" : "file:c:\\\\foo\\\\bar.html",
  "out" : "file:///C:/foo/bar.html"
}, {
  "in" : "  File:c|////foo\\\\bar.html",
  "out" : "file:///C:////foo/bar.html"
},{
  "in" : "http://look󠀠out.net/",
  "out" : "http://look%F3%A0%80%A0out.net/"
}, {
  "in" : "http://look־out.net/",
  "out" : "http://look%D6%BEout.net/"
},

Here is, how my encoding routine fails:

In:	http://www.example.com/##asdf
Expect:	http://www.example.com/##asdf
Actual:	http://www.example.com/#%23asdf

In:	http://www.example.com/#a
b
c	d
Expect:	http://www.example.com/#abcd
Actual:	http://www.example.com/#a%0Ab%0Dc%09d

In:	file:c:\\foo\\bar.html
Expect:	file:///C:/foo/bar.html
Actual:	ERROR

In:	  File:c|////foo\\bar.html
Expect:	file:///C:////foo/bar.html
Actual:	ERROR

java.net.URISyntaxException: Relative path in absolute URI: file://c:%5C%5Cfoo%5C%5Cbar.html
java.net.URISyntaxException: Relative path in absolute URI: file://c%7C////foo%5C%5Cbar.html
java.lang.IllegalArgumentException: java.text.ParseException: A prohibited code point was found in the inputlook󠀠out
In:	http://look󠀠out.net/
Expect:	http://look%F3%A0%80%A0out.net/
Actual:	ERROR

java.lang.IllegalArgumentException: java.text.ParseException: The input does not conform to the rules for BiDi code points.look־out
In:	http://look־out.net/
Expect:	http://look%D6%BEout.net/
Actual:	ERROR

Fazit

My  URL encoding routine needs still some refinement. Especially cases of double encoding and the handling of URL fragments must be subjects of further improvement. However I’m  already very happy with this standard Java solution. A more sophisticated approach can be found here: https://github.com/smola/galimatias and will also be subject of future tests.

Since this research is based on one of my stackoverflow answers, you can find the relevant code in my overflow repository.

Here is, how I encode URLs in Java.

1. Split URL into structural parts. Use java.net.URL for it.
2. Encode each part properly
3. Use IDN.toASCII(putDomainNameHere) to Punycode encode the host name!
4. Use java.net.URI.toASCIIString() to percent-encode, NFC encoded unicode – (better would be NFKC!). For more info see: How to encode properly this URL

   URL url= new URL("http://search.barnesandnoble.com/booksearch/first book.pdf);
   URI uri = new URI(url.getProtocol(), url.getUserInfo(), IDN.toASCII(url.getHost()), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
   String correctEncodedURL=uri.toASCIIString(); 
   System.out.println(correctEncodedURL);

   Prints

   http://search.barnesandnoble.com/booksearch/first%20book.pdf