How to convert Unicode URL to ASCII in Java

URLs can be quite complex when you add Internationalized Domain Names (IDNs) and Unicode characters to to the path. Often you’ll want to view and store these in ASCII, so proper conversion can become important. After much searching, I couldn’t find a great way to convert an entire Unicode URL to ASCII. Most examples just convert the domain to punycode, but forget about the port, path, and query string. Most examples don’t cover the case when the provided URL doesn’t have a scheme on the front. I tried to incorporate all of those URL components. I wanted a flexible conversion so I came up with some working code that probably has flaws, but it works for most URL formats you will encounter and a large variety that I tested it with.

Here’s a good list of domains to test this with. You can add ports, unicode paths, unicode params, and encoded paths characters to these for additional testing.
https://blogs.msdn.microsoft.com/shawnste/2006/09/14/idn-test-urls/

package com.company.utils;

import java.net.*;

public class UnicodeUtil {
    public static String convertUnicodeURLToAscii(String url) throws URISyntaxException {
        if(url != null) {
            url = url.trim();
            // Handle international domains by detecting non-ascii and converting them to punycode
            boolean isAscii = CharMatcher.ASCII.matchesAllOf(url);
            if(!isAscii) {
                URI uri = new URI(url);
                boolean includeScheme = true;

                // URI needs a scheme to work properly with authority parsing
                if(uri.getScheme() == null) {
                    uri = new URI("http://" + url);
                    includeScheme = false;
                }

                String scheme = uri.getScheme() != null ? uri.getScheme() + "://" : null;
                String authority = uri.getRawAuthority() != null ? uri.getRawAuthority() : ""; // includes domain and port
                String path = uri.getRawPath() != null ? uri.getRawPath() : "";
                String queryString = uri.getRawQuery() != null ? "?" + uri.getRawQuery() : "";

                // Must convert domain to punycode separately from the path
                url = (includeScheme ? scheme : "") + IDN.toASCII(authority) + path + queryString;

                // Convert path from unicode to ascii encoding
                url = new URI(url).toASCIIString();
            }
        }
        return url;
    }
}

Apache HTTP Client 4.3.4 Example with Timeouts, Client Cert and Trusted Cert from KeyStores

If you have ever used or tried to use the Apache HTTP Client library, you’ve probably discovered that is is very useful and powerful, but they change the library so much each version that it is hard to find proper examples of how to do complex things like adding connection timeouts, basic auth, or adding a client cert, or trusting a self-signed cert. Here’s a code snippet of getting a client cert and a trusted server cert loaded with timeouts in version 4.3.4 of the library.

    <dependency>
	<groupId>org.apache.httpcomponents</groupId>
	<artifactId>httpclient</artifactId>
	<version>4.3.4</version>
    </dependency>

    public static HttpClient getSSLClient(String serverKeystoreFile, String serverKeystorePassword, String serverKeystoreType,
                                         String clientCertFile, String clientCertPassword, String clientKeystoreType) throws Exception {

        // Server cert trust stuff
        KeyStore trustStore = KeyStore.getInstance(serverKeystoreType);
        trustStore.load(new FileInputStream(new File(serverKeystoreFile)), serverKeystorePassword.toCharArray());
        TrustManagerFactory trustFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustFactory.init(trustStore);
        TrustManager[] trustManagers = trustFactory.getTrustManagers();

        // Client cert stuff
        KeyStore clientCert = KeyStore.getInstance(clientKeystoreType);
        clientCert.load(new FileInputStream(new File(clientCertFile)), clientCertPassword.toCharArray());
        KeyManagerFactory keyFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        keyFactory.init(clientCert, clientCertPassword.toCharArray());
        KeyManager[] keyManagers = keyFactory.getKeyManagers();

        SSLContext sslcontext = SSLContext.getInstance("TLS");
        sslcontext.init(keyManagers, trustManagers, new SecureRandom());

        SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslcontext);

        HttpClient httpClient = HttpClients.custom()
                .setSSLSocketFactory(sslsf)
                .build();

        return httpClient;
    }

    public static String executePost() throws Exception {

	 URI uri = new URIBuilder("http://my.superawesome.fakeurlstuffandstuff.com/path1/path2/file.php")).build();
	 String serverKeystore = Properties.get("TRUSTED_CERT_KEYSTORE_FILE_PATH"); // JKS FILE
	 String serverPass = Properties.get("TRUSTED_CERT_KEYSTORE_PASSWORD");
	 String clientKeystore = Properties.get("CLIENT_KEYSTORE"); // PFX FILE
	 String clientPass = Properties.get("CLIENT_KEYSTORE_PASSWORD");

	 HttpClient client = HttpJsonUtil.getClient(serverKeystore, serverPass, "JKS", clientKeystore, clientPass, "PKCS12");

	 RequestConfig.Builder requestConfigBuilder = RequestConfig.custom();
	 requestConfigBuilder.setMaxRedirects(2);
	 requestConfigBuilder.setConnectionRequestTimeout(Properties.getInt("SERVER_REQUEST_TIMEOUT_MILLIS", 30_000));
	 requestConfigBuilder.setConnectTimeout(Properties.getInt("SERVER_CONNECT_TIMEOUT_MILLIS", 10_000));

	 httpPost = new HttpPost(uri);
	 httpPost.setConfig(requestConfigBuilder.build());
	 httpPost.setEntity(new StringEntity("REQUEST BODY"));
	 httpPost.setHeader("Content-Type", "application/json");
	 httpPost.setHeader("Accept", "application/json");
	 HttpResponse response = client.execute(httpPost);
	 return EntityUtils.toString(response.getEntity());       
    }

JVM proxy through a remote/reverse SSH tunnel

You may come across a scenario where you need to test your Java application from an alternate network, or you may not have sufficient network access where your code/application resides.  The JVM comes equipped with awesome proxy capabilities using SOCKs and a more specific HTTP proxy.  These can be set as environment properties when starting the JVM (http.proxyHost,http.proxyPort,socksProxyHost,socksProxyPort) or set programatically.  You can even configure a ProxySelector class for advanced proxy logic.  See Oracle’s Documentation for advanced usage.

Your http requests can go through SOCKs as well, but if the HTTPProxy is defined, then it will use the more specific proxy for the protocol it needs.

Here’s an example of the JVM options you would use to start your Java application to use proxies:

-Xmx2G -Dhttp.proxyHost=192.168.1.2 -Dhttp.proxyPort=80 -DsocksProxyHost=192.168.1.2 -DsocksProxyPort=1080

But, what if you don’t have direct access to the proxy server? You’ll need to use an SSH Tunnel. What if you can’t SSH out of your box? If you can SSH in, you can still create a remote/reverse tunnel. Here’s how that is done in Putty. Right click on the top of your active ssh putty window and chose ‘Change Settings’. Add the tunnels shown in the screenshots below to your connection. I noticed that sometimes ssh servers don’t have permissions to open up lower-numbered ports, so it took some trial and error before discovering that higher-numbered ports worked. The destination address of the tunnel is where the port will be forwarded to. The ‘Dynamic’ setting turns the tunnel into a SOCKs proxy itself and may be good if you don’t have a SOCKs server available, but I never got this working, but could have been because I was trying lower port numbers.

Putty1

Putty2

After you add those ports and apply the settings to your putty ssh session, your development machine now has two listening ports, one for http proxy, one for SOCKs proxy. Then you point your JVM at the localhost ports rather than the remote proxy ports.

-Xmx2G -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8555 -DsocksProxyHost=127.0.0.1 -DsocksProxyPort=8666

The SOCKs proxy in the JVM will even tell MySQL database connections to use the proxy. If you want a specific Apache HttpClient connection to use the proxy, you can configure it like this:

DefaultHttpClient client = new DefaultHttpClient();
HttpHost proxy = new HttpHost(“127.0.0.1”, 8555, “http”);
client.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);

or

client.getParams().setParameter(“socksProxyHost”,”127.0.0.1″);
client.getParams().setParameter(“socksProxyPort”, 8666);

If you are using Java’s URL class to make a request you can set a proxy for it:

Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(“127.0.0.1”, 8555));
HttpURLConnection connection = (HttpURLConnection) new
URL(myUrlString).openConnection(proxy);

If you are developing on a Windows box and need to SSH into that box, a good SSH Server that allows you to open up tunnels is COPSSH.

 

Java CAPTCHA w/ JCaptcha

When looking for a good Java CAPTCHA library, many people just suggested using ReCAPTCHA from Google.  A few online posts discouraged using ReCAPTCHA because it has shown swear words at times and has the possibility of being down which would also bring your own site down.  I like my Java Web Apps to have as few dependencies as possible, so I looked for a self-contained solution.  JCaptcha had everything I needed and was easy to integrate.  It was nice to be able to customize what kind of CAPTCHA words to use and what they would look like.

First off, add the dependency with Maven or as a Jar:

<dependency>
     <groupId>com.octo.captcha</groupId>
     <artifactId>jcaptcha</artifactId>
     <version>1.0</version>
 </dependency>

Add a custom engine to make it look how you want:

public class CustomCaptchaEngine extends ListImageCaptchaEngine {

     protected void buildInitialFactories() {

        com.jhlabs.image.WaterFilter water = new com.jhlabs.image.WaterFilter();
        water.setAmplitude(2d);
        water.setAntialias(true);
        water.setPhase(20d);
        water.setWavelength(70d);

        ImageDeformation backDef = new ImageDeformationByFilters(new ImageFilter[]{});
        ImageDeformation textDef = new ImageDeformationByFilters(new ImageFilter[]{});
        ImageDeformation postDef = new ImageDeformationByFilters(new ImageFilter[]{water});

        com.octo.captcha.component.word.wordgenerator.WordGenerator dictionaryWords = new com.octo.captcha.component.word.wordgenerator.ComposeDictionaryWordGenerator(
        new com.octo.captcha.component.word.FileDictionary("toddlist"));

        TextPaster randomPaster = new DecoratedRandomTextPaster(new Integer(6), new Integer(7), new SingleColorGenerator(Color.black), new TextDecorator[]{
        new BaffleTextDecorator(new Integer(1), Color.white)});
        BackgroundGenerator back = new UniColorBackgroundGenerator(new Integer(200), new Integer(75), Color.white);

        FontGenerator shearedFont = new RandomFontGenerator(new Integer(30), new Integer(35));

        com.octo.captcha.component.image.wordtoimage.WordToImage word2image;
        word2image = new DeformedComposedWordToImage(shearedFont, back, randomPaster, backDef, textDef, postDef);

        this.addFactory(new com.octo.captcha.image.gimpy.GimpyFactory(dictionaryWords, word2image));
    }
}

Then add an endpoint to serve up a BufferedImage.  I used Jersey for this:

public static ImageCaptchaService captchaService = new DefaultManageableImageCaptchaService(new FastHashMapCaptchaStore(), new CustomCaptchaEngine(), 180, 100000, 75000);

@Path("captcha.jpg")
@Produces("image/jpeg")
@GET
public Response lookupCategorization(@Context HttpServletRequest httpRequest) {

      BufferedImage bi = captchaService.getImageChallengeForID(httpRequest.getSession(true).getId());

 return Response.ok(bi)
      .header("Expires", 0)
      .header("Cache-Control", "no-store, no-cache, must-revalidate")
      .header("Pragma", "no-cache")
      .build();
 }

Add a validation method:

public static boolean validateResponse(HttpServletRequest request, String userCaptchaResponse) {
     if (request.getSession(false) == null) {
         return false;
     }
     boolean validResponse = false;
     try {
        validResponse = captchaService.validateResponseForID(request.getSession().getId(), userCaptchaResponse);
     } catch (CaptchaServiceException e) {}
     return validResponse;
}

Add a validation endpoint:

@Path("validate")
@Produces(MediaType.APPLICATION_JSON)
@POST
public Response validate(@Context HttpServletRequest httpRequest, @FormParam("captcha") String captcha) {

    if(validateResponse(httpRequest, captcha)) {
        ...
    } else {
        ...
    }
}

Add HTML form fields for CAPTCHA:

  <img class="captchaimage" src="captcha.jpg" />
  <input type="text" class="captchaText" autocomplete="off" name="captcha" /> 
  <a href="javascript: void(0);" onClick='loadNewCaptchaImage()'>Reload</a>

Add jQuery JavaScript to load a new image for users:

loadNewCaptchaImage = function () {
      $("img.captchaimage").attr("src", "captcha.jpg" + "?" + (new Date()).getTime()); 
      $('input.captchaText').val('').focus();
}

You can always make the captcha more difficult by adding more font variations, text effects, and background noise. Here are some examples of what you can do to make it difficult for bots to crack the captcha: https://jcaptcha.atlassian.net/wiki/display/general/Samples+tests  Just don’t frustrate your users by making it near impossible for them to figure out what the CAPTCHA says.

Remote deployment and debugging to Tomcat from IntelliJ including Remote JRebel updates.

Sometimes it’s nicer or required to develop code by deploying to a remote server. After a lot of trial and error, and frustration with lack of documentation on how this is to be done.  Here’s what I got to work. 

These instructions are mainly for windows, but can be adapted for other operating systems.

REMOTE TOMCAT BOX:

 Software needed:

 Delete the existing ROOT webapp folder in tomcat if you will be deploying to ‘/’

 Open up these ports in your firewall:

  • 1099
  • 8080
  • 5005 (Or whatever port you configure IntelliJ with for debugging)

 Add these lines to config/tomcat-users.xml:

<role rolename=”manager-gui” />

<role rolename=”manager-script” />

<role rolename=”manager-jmx” />

<role rolename=”manager-status” />

<user username=”admin” password=”password” roles=”manager-gui, manager-script, manager-jmx, manager-status” />

 Add setenv.bat (Windows) or setenv.sh to apache-tomcat/bin:

set CATALINA_OPTS=-Drebel.remoting_plugin=true -javaagent:C:\jrebel\jrebel.jar -Xdebug -Xrunjdwp:transport=dt_socket,address=5005,suspend=n,server=y -Xms64m -Xmx2048m -XX:MaxPermSize=768m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

 Update the javaagent param with the path to your installed jrebel.jar and the Xrunjdwp port with your IntelliJ debug port.

 Use startup.bat and shutdown.bat to control your tomcat instance.

LOCAL DEV BOX:

 Software needed:

  • IntelliJ 13
  • JRebel plugin
  • Maven 3

 Update your .m2/settings.xml file with:

<servers>

   <server>

      <id>RemoteTomcatServer</id>

      <username>admin</username>

      <password>password</password>

   </server>

</servers>

 Update the project’s main module’s pom.xml file with:

<plugin>

   <groupId>org.apache.tomcat.maven</groupId>

   <artifactId>tomcat7-maven-plugin</artifactId>

   <version>2.2</version>

   <configuration>

       <url>http://remotehost:8080/manager/text</url>

       <server>RemoteTomcatServer</server>

       <path>/</path>

   </configuration>

</plugin>

 With this maven plugin, you can run ‘mvn tomcat7:deploy’, ‘mvn tomcat7:redeploy’, etc to build and send war files to the remote tomcat instance.  You can do this from the command line or the Maven Projects tab in IntelliJ.

 Create an IntelliJ ‘Remote’ run configuration with your debugging port information and add a maven command to the ‘Before launch’ box with tomcat7:redploy as the command in the project’s main module directory.  You can have these actions in separate run configurations if you want the debugging and deployment to be executed as needed.

 IntelliJ JRebel settings:

File->Settings->Compiler->  check ‘Make project automatically

File->Project Settings->Modules->  enable ‘Remote Host’ in the modules’ JRebel configuration with your remote host url (http://remotehost:8080/)

To send changed resources, jsps, and class files over to the remote server: Run->JRebel Remoting: Synchronize project

 References:

http://wiki.aiwsolutions.net/2014/02/20/deploy-web-application-to-remote-tomcat- 7-server/

http://tomcat.apache.org/maven-plugin-2.0/tomcat7-maven-plugin/usage.html

http://zeroturnaround.com/software/jrebel/learn/remoting/setting-up-jrebel-remot ing-with-intellij-idea-and-tomcat/

http://player.vimeo.com/video/54855437

http://rockycode.com/blog/debugging-remote-tomcat-intellij/

 

Play Framework 2.0 Java Controller Route Sub-Packages/Folders

The Play Framework is an awesome change from JavaEE. I like how they embrace Object Oriented design throughout the framework. The Play Framework is Java’s version of Django or Ruby on Rails. By default, it uses EBean as the ORM and it has been a refreshing change from Hibernate as well. It is the the most rapid web development framework that I have used in Java. I came across my first curve-ball with it that was not easy to find when googling it.

I don’t like flat project structures. I always plan that even the smallest projects will have a ton of models, controllers, and views and they should all be segregated in their own folders/packages. I placed my logged-in controllers in their own package in controllers.tracker.*. The trick was to get the routes object to work.

I assumed that routes.tracker.DashboardController.index() would work, nope. Apparently there is a routes object at each level of the package structure under the controllers package. So here’s what worked:

    return redirect(controllers.tracker.routes.DashboardController.index());

Geolocating and Mapping Points with Nominatim and Leaflet

I wanted to add a map feature to http://www.shelfreliancepro.com but had no budget to pay for map apis or libraries. The solution I found was http://nominatim.openstreetmap.org and http://leafletjs.com/.  Nominatim allows you to do all kind of geo-locating, address queries, and location queries.  Leaflet is a simple map interface for plotting points.  Leaflet works very well on mobile devices.

When users would update or add an address to the system, I would mark the address so I knew it needed to have it’s latitude and longitude updated via nominatim.  I run a scheduled task every 10 minutes to find addresses that need an update and update them.

Here’s how it turned out:

map