hdfs app 프로그램

1. 코드 작성

import java.io.InputStream;
import java.net.URL;
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;
public class FileRead {
    static {
        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
    }
    public static void main(String[] args) throws Exception {
        InputStream fileInputStream = null;
        try {
            fileInputStream = new URL(args[0]).openStream();
            IOUtils.copyBytes(fileInputStream, System.out, 4096, false);
        } finally {
            IOUtils.closeStream(fileInputStream);
        }
    }
}

 

2. compile 및 실행

cdh 기준으로 작성

 

1) compile

javac -cp `hadoop classpath`  FileRead.java -d build -Xlint

 

2) jar packing

jar -cvf file_read.jar -c build/ .

 

3) 실행

export HADOOP_CLASSPATH=file_read.jar

hadoop FileRead hdfs://cluster1.test.co.kr:8020/tmp/test.csv