Sunday, September 21, 2014

JSR 199 - Compiler API

JSR 199 provides the compiler API to compile the Java code inside another Java program. The following are the important classes and interfaces provided for facilitating the compilation from a Java program.
  • JavaFileObject - Represents a compilation unit, typically a class source.
  • SimpleJavaFileObject - Implementation of the methods defined in JavaFileObject
  • DiagnosticCollector - Collects the compilation errors, warning into a list of Diagnostic type
  • Diagnostic - Reports the type of the problem and details like line number, character, error reason etc. 
  • JavaFileManager - To work on the Java source and class files.
  • JavaCompiler - The compiler instance for compiling the compilation unit. 
  • CompilationTask - A sub interface of JavaCompiler which helps to compile and return the status with diagnostic when used call method on it. 

Where to start

To compile a Java code, we need the Java source. The source can be a physical file on the disk or a string inside the program. Using the source, we need create an instance type of JavaFileObject.

Using String literal

Create a class which implements JavaFileObject, here i am using SimpleJavaFileObject. We need create the path URI of the class file
package com.test;

import java.io.IOException;
import java.net.URI;

import javax.tools.SimpleJavaFileObject;

public class SampleSource extends SimpleJavaFileObject
{ 
    private String source;

    protected SampleSource(String name, String code) {
        super(URI.create("string:///" +name.replaceAll("\\.", "/") + Kind.SOURCE.extension), Kind.SOURCE);
        this.source = code ;
    }
 
    @Override
    public CharSequence getCharContent(boolean ignoreEncodingErrors)
            throws IOException {
        return source ;
    }
}
Now, create the instance of JavaFileObject and from those, create the Compilation Unit (A collection of JavaFileObject)
String str = "package com.test;"
                + "\n" + "public class Test {"
                + "\npublic static void test() {"
                + "\nSystem.out.println(\"Comiler API Test\")-;" + ""
                        + "\n}" + "\n}";

        SimpleJavaFileObject fileObject = new SampleSource("com.test.Test", str);
        JavaFileObject javaFileObjects[] = new JavaFileObject[] { fileObject };
        Iterable<? extends JavaFileObject> compilationUnits = Arrays
                .asList(javaFileObjects);

From File System

If the source is from physical location. Then create like this.
File []files = new File[]{file1, file2, file3, file4} ;
Iterable<? extends JavaFileObject> units =
           fileManager.getJavaFileObjectsFromFiles(Arrays.asList(files));

Create a JavaFileManger

We will see, how to create a fileManger now.
JavaFileManager fileManager = compiler.getStandardFileManager(
                diagnostics, Locale.getDefault(), Charset.defaultCharset());
To get the FileManger, we need
  • diagnostic - A DiagnosticCollector of JavaFileObject
  • locale - The locale of the compilation
  • charset - The charset to be used.

Compiler

Get the compiler instance using ToolProvider. Finally, create the CompilationTask from the compiler instance using diagnostics, file manager and compilation units (Optionally writer and compilation options).
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
CompilationTask task = compiler.getTask(null, fileManager, diagnostics,
                compilationOptionss, null, compilationUnits);
The argument required to get the CompilationTask are
  • out - A writer which writes the output of the compiler. Defaults to System.err if null 
  • listener - A diagnostic listener, the errors or warning can be accessed using.
  • options - Compiler options (Ex : -d, like we give in command line using javac ) 
  • classes - Name of the classes to be processed 
  • compilationUnits - List of compilation units

Compile

Finally, call the method to compile. This method to be called only once otherwise it throws IllegalStateException on multiple calls. Once compiled, returns true for successful compilation otherwise false. We need to look the diagnosticCollector to get the error/warning details.
boolean status = task.call();

All together

Putting all together.

    public static void main(String[] args)
    {
        String str = "package com.test;"
                + "\n" + "public class Test {"
                + "\npublic static void test() {"
                + "\nSystem.out.println(\"Comiler API Test\")-;" + ""
                        + "\n}" + "\n}";

        SimpleJavaFileObject fileObject = new SampleSource("com.test.Test", str);
        JavaFileObject javaFileObjects[] = new JavaFileObject[] { fileObject };
        Iterable<? extends JavaFileObject> compilationUnits = Arrays
                .asList(javaFileObjects);

        Iterable<String> compilationOptionss = Arrays.asList(new String[] {
                "-d", "classes" });
        
        DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();

        JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();

        JavaFileManager fileManager = compiler.getStandardFileManager(
                diagnostics, Locale.getDefault(), Charset.defaultCharset());
        CompilationTask task = compiler.getTask(null, fileManager, diagnostics,
                compilationOptionss, null, compilationUnits);
        boolean status = task.call();
        
        if(!status)
        {
            System.out.println("Found errors in compilation");
            int errors = 1;
            for(Diagnostic diagnostic : diagnostics.getDiagnostics())
            {
                printError(errors, diagnostic);
                errors++;
            }
        }
        else
            System.out.println("Compilation sucessfull");
        
        try
        {
            fileManager.close();
        } catch (IOException e){}

    }
    
    public static void printError(int number,Diagnostic diagnostic)
    {
        System.out.println();
        System.out.print(diagnostic.getKind()+"  : "+number+" Type : "+diagnostic.getMessage(Locale.getDefault()));
        System.out.print(" at column : "+diagnostic.getColumnNumber());
        System.out.println(" Line number : "+diagnostic.getLineNumber());
        System.out.println("Source : "+diagnostic.getSource());
        
    }

Output

Output with an error will be (because of an hyphen in System.out.println in main method of Test)
Found errors in compilation

ERROR  : 1 Type : illegal start of expression at column : 40 Line number : 4
Source : com.test.SampleSource[string:///com/test/Test.java]

ERROR  : 2 Type : not a statement at column : 39 Line number : 4
Source : com.test.SampleSource[string:///com/test/Test.java]
To read more about JSR 199, follow the official link.

Happy Learning!!!! 

Monday, September 15, 2014

Java Heap Memory

This is continuation of the previous post. As i mentioned earlier, the Java Memory is divided into multiple generations for the performance of JVM as explained below.
  • Permanent Generation
  • Old or Tenured Generation
  • Young Generation


Each of the above spaces (with minimum and maximum values) can be configurable through the command line parameters. If not configured, JVM defaults each of these based on the mode of JVM and the platform running. Initial size is the size allocated during the startup of JVM irrespective of the usage of the application and the remaining size will be reserved. When application is running out of the initial size, then it starts using the remaining space until the the maximum size is reached. Once the size reached maximum and memory can't be allocated, it throws an OutOfMemoryError exception. The total size of the heap is configured using
  • -Xms: The minimum size of the heap space
  • -Xmx: The maximum size of the heap space

Permanent Generation

Permanent generation is the part of the heap memory where the class meta data and string constants are stored. This is also one of the space which is garbage collected. Most of us (including me) think that Permanent generation is created and never GCed. GC will run on the permanent generation when there are no references on the constants or the constants are never used. The Permanent Generation space is configurable using the following two parameters
  • -XX:PermSize: Initial permanent generation size during the start of VM.
  • -XX:MaxPermSize: This is the maximum size of the permanent generation which can be expanded up-to from initial size.

Old Generation

As the name indicates, this is the pool of the objects which are survived during the full GCs. These are the objects copied from the survivor spaces (Will see this very shortly). The size of the old generation is always relative to the young generation. The parameters will be explained in conjunction with Young Generation

Young Generation

The new objects are always created in the Eden space of the Young Generation. When GC happens, the objects which are survived will be moved to the Survivor space 1 (From). Subsequently, the objects survived in the From Survivor Space will be moved to To Survivor Space. So, in short the objects are initially created in Eden and survived objects will be moved to From and To Survivor spaces and Eden space is cleared to create more new objects.
The size of the survivor space can be set relative to the Eden Space using the option -XX:SurvivorRatio. The sizes of two survivor spaces will be same. So, if we set-XX:SurvivorRatio=6, each survivor space will be one-eight of the young generation and Eden will be six-eight. 
The parameters -XX:NewSize and -XX:MaxNewSize are the initial and maximum sizes of the Young generation. The ratio between Young and Old generation is set using -XX:NewRatio. If we set -XX:NewRatio=3, then the total young generation (both Eden and Survivor space) will be one-third of the total heap size. 

Command line parameters

  • -Xms size: The minimum size of the Heap Space.
  • -Xmx size: The maximum size of the Heap Space.
  • -XX:NewSize=size: The initial size of the Young Generation.
  • -XX:MaxNewSize=size: The maximum size of the Young Generation.
  • -XX:SurvivorRatio=ratio: The ratio between survivor space and the Eden space.
  • -XX:NewRatio=ratio: The ratio between Young and Old Generations.
  • -XX:+AggressiveHeap: This parameters let the JVM to allocate the as much space as possible until the space is available on the machine. This is one of the GC tuning option to be set for long running programs. (May be for the application servers).

Points to remember

  • If the NewRatio is high, then the time between full GCs will be more
  • If the survivor spaces are tool small, then the objects will be directly moved to Old Generation without using the Survivor in case of Large objects.
  • The young and old generations are always related to minimum and maximum sizes of the Heap. 
In the next post, we will see more details on the GC and moving objects between different generations.

Happy Learning!!!!

Saturday, September 13, 2014

Java Memory Management - Overview

JVM creates various run time areas during the startup for the execution of the program. The following figure illustrates the memory model used by JVM.

JVM creates

  • Method Area
  • Native Method Stack
  • Java Stack
  • pc Register
  • Heap Space


Method Area

Method area is the space where the compiled code of all the JVM Threads contain. It stores the data common to all the threads like methods, constructors of each class, common run time variables etc. Its very similar to  Heap memory but this area couldn't get compact or clean up during the garbage collection. It is created during the startup of JVM.
  • If area is not able to allocate the request, then JVM throws an OutOfMemoryError.

Native Method Stack

This area is used to support the conventional native methods for invoke operation. This memory is for the methods written in other than Java. This area is created by the creation of the Thread.
  • If native methods require a large amount of stack than supplied, it raises StackOverflowError
  • If memory can be extended but couldn't able to allocate, it raises OutOfMemoryError

Java Stack

In other words, its JVM private method stack. This is same as Conventional method stack but used for only methods created in Java.
  • If methods require a large amount of stack than supplied, it raises StackOverflowError
  • If memory can be extended but couldn't able to allocate, it raises OutOfMemoryError

pc Register

As Java can execute multi-threads, each thread has it's own pc register to store the current instruction being executed by the thread. It contains the value only the thread is executing a non-native instruction. If the thread is executing a native instruction, it will be undefined.

Heap

Heap is the common run time work area for all the threads. It stores the local variables, collections, objects data being used by the each thread.  The reclaimed objects will be collected/removed by the automatic memory management process called Garbage Collector. The head size can be fixed by using the command line parameters otherwise it's not limited (depends on the system where the JVM is running).
To increase the performance of the JVM, the heap is divided into multiple generation as you see in the above picture. So Heap is divided into Young generation and Old or Tenured Generation and Permanent Generation. 

PermGen Space

PermGen space will be used by JVM to keep the loaded class and other common spaces like String pool. This space is directly proportional to the size of the application (classes, strings created etc.) and also GC cannot do much clean-up on this, so it will easily run out of memory with PermGen out of space error if not defined properly. 

Heap and PermGen space can be configurable using command line arguments while starting the JVM. In the next sections, will see more details on the Heap Memory, PermGen Space and Garbage Collector and their settings. 

Wednesday, September 10, 2014

CSV Operations using OpenCSV

OpenCSV is one of the best tools for CSV operations. We will see how to use OpenCSV for basic reading and writing operations.

How to use

Maven dependency for opencsv is as below
     <dependency>
	<groupId>net.sf.opencsv</groupId>
	<artifactId>opencsv</artifactId>
	<version>2.3</version>
     </dependency>

What it provides

  • CSVReader: Provides the operations to read the CSV file as a list of String array. It can used along with CsvToBean to read the data as list of beans
  • CSVWriter: Allows us to write the data to a CSV file.
  • CsvToBean: Reads the CSV data and coverts to a bean using MappingStrategy.
  • MappingStrategy: It's an interface to define the mapping between the data being written to the header of the CSV. It's been implemented by HeaderColumnNameMappingStrategy,  ColumnPositionMappingStrategy and HeaderColumnNameTranslateMappingStrategy for respective mapping formats. 

Writing to a CSV file

See below an example of writing a CSV file using CSVWriter

        String csv = "/tmp/employee.csv";
        CSVWriter writer = new CSVWriter(new FileWriter(csv));
         
        String[] header= new String[]{"Name","Age","Salary","Address"};
        writer.writeNext(header);

        List<String[]> allData = new ArrayList<String[]>();
        for(int i=0;i<3;i++)
        {
            String[] data = new String[]{"Blogger"+i,"20"+i,"20.0002",i+" World Wide Web"};
            allData.add(data);
        }
        
        writer.writeAll(allData);
        writer.close();

Points to be noted

  • CSVWriter can write line by line using writeNext
  • writeAll can used to write full CSV data at once.
  • By default, the separator will be a comma(,). If you want to make another character as a separator we can pass it as an argument.
  • The maximum possible signature of the constructor is :
    public CSVWriter(Writer writer, char separator, char quotechar, char escapechar)

Output

"Name","Age","Salary","Address"
"Blogger0","200","20.0002","0 World Wide Web"
"Blogger1","201","20.0002","1 World Wide Web"
"Blogger2","202","20.0002","2 World Wide Web"

Reading from a CSV File

Reading a csv can be done in two ways. One as a list of string array or as a bean.

Reading as an array

        CSVReader csvReader = new CSVReader(new FileReader("/tmp/employee.csv"));
        List<String[]> allData = csvReader.readAll();
        
        for(String[] data : allData)
        {
            for(String s : data)
            {
                System.out.print(s+";");
            }
            System.out.println();
        }
        
        csvReader.close();

Points to be noted

  • As CSVWriter, CSVReader also provides readNext and readAll methods to read one line or full data respectively and the delimiter can be specified while reading the file (Other than comma(,)). 
  • Adding to that, we can set the ignore spaces, default skip lines, special quote character etc.. while reading a CSV file
  • When we read as an array of string, header will not ignored. So we need to skip the first element in the list or we can specify start line while creating CSVReader. See the output below

Output

Name;Age;Salary;Address;
Blogger0;200;20.0002;0 World Wide Web;
Blogger1;201;20.0002;1 World Wide Web;
Blogger2;202;20.0002;2 World Wide Web;

Reading as a Bean

The CSV data can be read into a bean, but we need to define the mapping strategy and pass the strategy to CsvToBean to parse the data into a bean.
        Map<String, String> mapping = new HashMap<String, String>();
        mapping.put("Name", "name");
        mapping.put("Age", "age");
        mapping.put("Salary", "salary");
        mapping.put("address", "Address");
        
        HeaderColumnNameTranslateMappingStrategy<Employee> strategy = new HeaderColumnNameTranslateMappingStrategy<Employee>();
        strategy.setType(Employee.class);
        strategy.setColumnMapping(mapping);
        
        CSVReader csvReader = new CSVReader(new FileReader("/tmp/employee.csv"));
        CsvToBean<Employee> csvToBean = new CsvToBean<Employee>();
        List<Employee> list = csvToBean.parse(strategy, csvReader);
        
        for(Employee e : list)
        {
            System.out.println(e);
        }

Points to be noted 

  • Here we used HeaderColumnNameTranslateMappingStrategy which maps the column id to the bean property. 
  • CSV Reader and strategy to be passed to CsvToBean to read the data into the bean. When we parse, we get the list of bean as a result.

Output

Name : Blogger0; Age : 200; Salary : 20.0002; Addresss : 0 World Wide Web
Name : Blogger1; Age : 201; Salary : 20.0002; Addresss : 1 World Wide Web
Name : Blogger2; Age : 202; Salary : 20.0002; Addresss : 2 World Wide Web
Happy Learning!!!!

Sunday, September 7, 2014

Resource Pooling in Java

Resource pooling is the most important when we are dealing with multi-threaded, concurrent applications or multi-tier application which deal with limited resources to effectively manage them. We have APIs for pooling different resources like database connections, messages processing etc. Here, we see very simple object pooling with an example using Apache Common Pooling (ACP).

Context

A multi-threaded application accessing an resource (limited - only 3 in this case). This takes a bit long time to complete the action. I will simulate the case with a thread which takes 8s to execute.
package com.test.thread;

public class MyThread implements Runnable
{
    private boolean running = false;

    public void run()
    {
        int time = 4;   
        while(time > 0)
        {
            try
            {
                Thread.sleep(2000);
            } catch (InterruptedException e)
            {
                e.printStackTrace();
            } 
            time--;
        }
    }

    public boolean isRunning()
    {
        return running;
    }

    public void setRunning(boolean running)
    {
        this.running = running;
    }
}

Without Pooling

Consumer

The consumer uses this resources for completing a task. These are real-time consumers (not countable) which access the resource at any time.
package com.test.thread;

public class Consumer implements Runnable
{

    int i;

    public Consumer(int i)
    {
        this.i = i;
    }
    public void run()
    {
        MyThread m = null;
        try
        {
            m = new MyThread();
            System.out.println("Started " + i);
            Thread t = new Thread(m);
            t.start();
            while (t.isAlive());
            System.out.println("Completed " + i);
        } catch (Exception e)
        {
            System.out.println("Exception while getting the object " + i);
            e.printStackTrace();
        }
    }
}

Application

Assume that at any point of time we have 10 consumers accessing the MyThread. As one consumer is using one MyThread, there will be 10 MyThreads' created.
package com.test.thread.appl;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import com.test.thread.Consumer;
import com.test.thread.MyThread;

public class Main
{

    public static void main(String[] args) throws Exception
    {
        ExecutorService e = Executors.newFixedThreadPool(10);
        List<Future> f = new ArrayList<Future>();
        
        for(int i=0;i<10;i++)
        {
            System.out.println("Getting the object "+i);
            Future a = e.submit(new Consumer(i));
            f.add(a);
        }
        
        label: for(Future s : f)
        {
            if(!s.isDone())
                continue label;
        }
    }
}

With Pooling

This will work as long as we don't have a limit on number of MyThread. Suppose if we limit number of MyThread to 3. Our application may run into problems because of resource shortage. To effectively handle this, we use Resource Pooling. There will be no change in the Consumer except that it gets the object from the pool instead of creating.

Pool Factory

package com.test.thread;

import org.apache.commons.pool2.BasePooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;

public class MyThreadFactory extends BasePooledObjectFactory<MyThread>
{

    @Override
    public MyThread create() throws Exception
    {
        return new MyThread();
    }

    @Override
    public PooledObject<MyThread> wrap(MyThread arg0)
    {
        return new DefaultPooledObject<MyThread>(arg0);
    }
    
    @Override
    public boolean validateObject(PooledObject<MyThread> p)
    {
        return p.getObject().isRunning();
    }
    
    @Override
    public void activateObject(PooledObject<MyThread> p) throws Exception
    {
        p.getObject().setRunning(true);
    }
    
    @Override
    public void passivateObject(PooledObject<MyThread> p) throws Exception
    {
        p.getObject().setRunning(false);
    }

}

Consumer

Now change the consumer to load MyThread from pool.
package com.test.thread;

import org.apache.commons.pool2.impl.GenericObjectPool;

public class Consumer implements Runnable
{

    GenericObjectPool<MyThread> pool;
    int i;

    public Consumer(int i, GenericObjectPool<MyThread> pool)
    {
        this.i = i;
        this.pool = pool;
    }

    public void run()
    {
        MyThread m = null;
        try
        {
            m = pool.borrowObject();
            System.out.println("Started " + i);
            Thread t = new Thread(m);
            t.start();
            while (t.isAlive());
            System.out.println("Completed " + i);
        } catch (Exception e)
        {
            System.out.println("Exception while getting the object " + i);
            e.printStackTrace();
        }
        if (m != null)
            pool.returnObject(m);
    }
}

Points to remember: 

  • We need to get the object from the pool using borrowObject. 
  • When we finished using, it should be returned to the pool using returnObject. Otherwise we may run into resource unavailability. 
  • borrowObject can wait for specific time if the resource is not free. borrowObject(millis)

Application

package com.test.thread.appl;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import org.apache.commons.pool2.impl.GenericObjectPool;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;

import com.test.thread.Consumer;
import com.test.thread.MyThread;
import com.test.thread.MyThreadFactory;

public class Main
{

    public static void main(String[] args) throws Exception
    {
        GenericObjectPoolConfig config = new GenericObjectPoolConfig();
        config.setMaxTotal(3);
        config.setBlockWhenExhausted(true);
        config.setMaxWaitMillis(30 * 1000);
        
        MyThreadFactory factory = new MyThreadFactory();
        GenericObjectPool<MyThread> pool = new GenericObjectPool<MyThread>(factory, config);
        
        ExecutorService e = Executors.newFixedThreadPool(10);
        List<Future> f = new ArrayList<Future>();
        
        for(int i=0;i<10;i++)
        {
            System.out.println("Getting the object "+i);
            Future a = e.submit(new Consumer(i,pool));
            f.add(a);
        }
        
        label: for(Future s : f)
        {
            if(!s.isDone())
                continue label;
        }
    }

}

Points to remember

  • Configuration of the pool can be set using GenericObjectPoolConfig (max Total, max time to wait etc.) 
  • Factory is used to retrieve and put back the resource. 
  • activateObject is called once the object is retrieved from the pool 
  • passivateObject reset the object after the object returns to the pool. 
  • Pool validates the object each time before fetching it. 
  • If all the resources are busy, GenericObjectPool waits for maxWaitMillis to get a resource. If not, throws an exception. The pool can be blocked by setting : blockWhenExhausted

Execution

When we run the above example, GenericObjectPool will block after 3 MyThread's are created. Once one of them returns to the object, will be resumed.
Getting the object 0
Getting the object 1
Getting the object 2
Getting the object 3
Getting the object 4
Getting the object 5
Getting the object 6
Getting the object 7
Getting the object 8
Getting the object 9
Started 0
Started 1
Started 2
Completed 0
Started 3
Completed 1
Completed 2
Started 5
Started 4
Exception while getting the object 6
Exception while getting the object 8
Exception while getting the object 7
Exception while getting the object 9
java.util.NoSuchElementException: Timeout waiting for idle object
 .....
Completed 3
Completed 5
Completed 4

Points to remember

When we look at the output, 
  • Only 3 objects were returned from the pool and it's blocked while getting next. 
  • Once one of them is completed, next is started. 
  • If the wait time is completed and no object found, throws an exception NoSuchElementException 
  • Most important point is we should clearly identify the maximum wait time when we are writing
Please read through the javadoc and examples for more details on the official site.

Happy Learning!!!