{"id":157,"date":"2018-11-15T13:39:13","date_gmt":"2018-11-15T13:39:13","guid":{"rendered":"https:\/\/jan.schnasse.org\/blog\/?p=157"},"modified":"2018-11-15T13:39:13","modified_gmt":"2018-11-15T13:39:13","slug":"url-encoding-in-java-explained","status":"publish","type":"post","link":"https:\/\/jan.schnasse.org\/blog\/2018\/11\/15\/url-encoding-in-java-explained\/","title":{"rendered":"URL encoding in Java – Explained"},"content":{"rendered":"

Recently I introduced a solution for URL encoding in Java .<\/a><\/p>\n

public static String encode(String url) {\n    try {\n      URL u = new URL(url);\n      URI uri = new URI(u.getProtocol(), \n                        u.getUserInfo(), \n                        IDN.toASCII(u.getHost()), \n                        u.getPort(), \n                        u.getPath(),\n                        u.getQuery(), \n                        u.getRef());\n      String correctEncodedURL = uri.toASCIIString();\n      return correctEncodedURL;\n    } catch (Exception e) {\n      throw new RuntimeException(e);\n    }\n}<\/pre>\n
 <\/p>\n
Now I like to introduce a set of URLs to test the code.\u00a0 Good test sets are provided at the\u00a0 ‚Web Platform Tests‘ (wpt)<\/a> repository.\u00a0 A comprehensible assembly of infos about the URL standard can be found at whatwg.org.<\/a><\/p>\n
On basis of the ‚Web Platform Tests‘ I created a file to hold test urls<\/a> together with the expected outcome. The test set is provided in the following form:<\/p>\n
{ \n  \"in\" : \"http:\/\/\u4f60\u597d\u4f60\u597d.urltest.lookout.net\/\",\n  \"out\" : \"http:\/\/xn--6qqa088eba.urltest.lookout.net\/\" \n}, \n{ \n  \"in\" : \"http:\/\/urltest.lookout.net\/?q=\\\"asdf\\\"\", \n  \"out\" : \"http:\/\/urltest.lookout.net\/?q=%22asdf%22\" \n}<\/pre>\n
To test my URL encoding implementation I use the following code<\/p>\n
try (InputStream in = Thread.currentThread().getContextClassLoader()\n        .getResourceAsStream(\"url-succeding-tests.json\")) {\n  ObjectMapper mapper = new ObjectMapper();\n  JsonNode testdata = mapper.readValue(in, JsonNode.class).at(\"\/tests\");\n  for (JsonNode test : testdata) {\n    String url = test.at(\"\/in\").asText();\n    String expected = test.at(\"\/out\").asText();\n    String encodedUrl = URLUtil.encode(url);\n    org.junit.Assert.assertTrue(expected.equals(encodedUrl));\n  }\n} catch (Exception e) {\n  throw new RuntimeException(e);\n}<\/pre>\n
During my tests I also found some URLs that were not encoded correctly. I collected them in another JSON file<\/a>.<\/p>\n
Here are some failing examples:<\/p>\n
{\n  \"in\" : \"http:\/\/www.example.com\/##asdf\",\n  \"out\" : \"http:\/\/www.example.com\/##asdf\"\n},{\n  \"in\" : \"http:\/\/www.example.com\/#a\\nb\\rc\\td\",\n  \"out\" : \"http:\/\/www.example.com\/#abcd\"\n}, {\n  \"in\" : \"file:c:\\\\\\\\foo\\\\\\\\bar.html\",\n  \"out\" : \"file:\/\/\/C:\/foo\/bar.html\"\n}, {\n  \"in\" : \"  File:c|\/\/\/\/foo\\\\\\\\bar.html\",\n  \"out\" : \"file:\/\/\/C:\/\/\/\/foo\/bar.html\"\n},{\n  \"in\" : \"http:\/\/look\udb40\udc20out.net\/\",\n  \"out\" : \"http:\/\/look%F3%A0%80%A0out.net\/\"\n}, {\n  \"in\" : \"http:\/\/look\u05beout.net\/\",\n  \"out\" : \"http:\/\/look%D6%BEout.net\/\"\n},<\/pre>\n
Here is, how my encoding routine fails:<\/p>\n
In:\thttp:\/\/www.example.com\/##asdf\nExpect:\thttp:\/\/www.example.com\/##asdf\nActual:\thttp:\/\/www.example.com\/#%23asdf\n\nIn:\thttp:\/\/www.example.com\/#a\nb\nc\td\nExpect:\thttp:\/\/www.example.com\/#abcd\nActual:\thttp:\/\/www.example.com\/#a%0Ab%0Dc%09d\n\nIn:\tfile:c:\\\\foo\\\\bar.html\nExpect:\tfile:\/\/\/C:\/foo\/bar.html\nActual:\tERROR\n\nIn:\t  File:c|\/\/\/\/foo\\\\bar.html\nExpect:\tfile:\/\/\/C:\/\/\/\/foo\/bar.html\nActual:\tERROR\n\njava.net.URISyntaxException: Relative path in absolute URI: file:\/\/c:%5C%5Cfoo%5C%5Cbar.html\njava.net.URISyntaxException: Relative path in absolute URI: file:\/\/c%7C\/\/\/\/foo%5C%5Cbar.html\njava.lang.IllegalArgumentException: java.text.ParseException: A prohibited code point was found in the inputlook\udb40\udc20out\nIn:\thttp:\/\/look\udb40\udc20out.net\/\nExpect:\thttp:\/\/look%F3%A0%80%A0out.net\/\nActual:\tERROR\n\njava.lang.IllegalArgumentException: java.text.ParseException: The input does not conform to the rules for BiDi code points.look\u05beout\nIn:\thttp:\/\/look\u05beout.net\/\nExpect:\thttp:\/\/look%D6%BEout.net\/\nActual:\tERROR<\/pre>\n
Fazit<\/h2>\n
My\u00a0 URL encoding routine<\/a> needs still some refinement. Especially cases of double encoding and the handling of URL fragments must be subjects of further improvement. However I’m\u00a0 already very happy with this standard Java solution. A more sophisticated approach can be found here: https:\/\/github.com\/smola\/galimatias<\/a> and will also be subject of future tests.<\/p>\n
Since this research is based on one of my stackoverflow answers<\/a>, you can find the relevant code in my overflow repository<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Recently I introduced a solution for URL encoding in Java . public static String encode(String url) { try { URL u = new URL(url); URI uri = new URI(u.getProtocol(), u.getUserInfo(), IDN.toASCII(u.getHost()), u.getPort(), u.getPath(), u.getQuery(), u.getRef()); String correctEncodedURL = uri.toASCIIString(); return correctEncodedURL; } catch (Exception e) { throw new RuntimeException(e); } }   Now I like […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,9,15],"tags":[],"class_list":["post-157","post","type-post","status-publish","format-standard","hentry","category-development","category-java","category-software"],"_links":{"self":[{"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/posts\/157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/comments?post=157"}],"version-history":[{"count":0,"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/posts\/157\/revisions"}],"wp:attachment":[{"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/media?parent=157"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/categories?post=157"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jan.schnasse.org\/blog\/wp-json\/wp\/v2\/tags?post=157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}