前言
在Hue上面测试一个很简单的Sqoop
import--connectjdbc:mysql://10.169.xx.xxx/test--usernameroot--passwordroot--as-textfile--columnsid,name,english,chinese,math--tableexam--target-dirhdfs://master02:8020/tmp/test--check-columnid--incrementalappend-m1
这个代码是在本地测试通过的,我本认为在Hue上可以很轻松的通过,没想到直接报错:
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
一看这个bug,刚开始觉得很简单,只要Sqoop获取到jdbc驱动就可以了,可是这个jdbc驱动应该往哪里放啊?还有为什么本地测试竟然可以,而Hue就不行呢?郁闷ing。。。
究竟往哪里放JDBC驱动包呢
往每个节点的/CDH/lib/sqoop/lib下面都放往每个节点/var/lib/sqoop下面都放
往HDFS的/user/oozie/share/lib/lib_timstamp/sqoop目录下放
不对啊,放在/user/oozie/share/lib/lib_timstamp/sqoop目录下后得重启oozie服务啊
终于成功了…
后记
其实就两步:
1. 将mysql-connector-java.jar上传到hdfs:/user/oozie/share/lib/lib_timstamp/sqoop2, 重启oozie服务。
还有,在hue的页面上无论出现什么错,总是报如下错误迷惑人:
>>> Invoking Sqoop command line now >>>19:51:10.724 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.<<< Invocation of Sqoop command completed <<<No child hadoop job is executed.java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)Caused by: java.lang.SecurityException: Intercepted System.exit(1)at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)at java.lang.Runtime.exit(Runtime.java:107)at java.lang.System.exit(System.java:971)at org.apache.sqoop.Sqoop.main(Sqoop.java:252)at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:214)at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:199)at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104)at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)... 16 moreIntercepting System.exit(1)Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]Oozie Launcher, uploading action data to HDFS sequence file: hdfs://nameservice2/user/xuec7/oozie-oozi/0000000-191219193920246-oozie-oozi-W/sqoop-213e--sqoop/action-data.seq19:51:10.778 [main] INFO org.apache.press.CodecPool - Got brand-new compressor [.deflate]Stopping AM19:51:10.801 [main] INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Waiting for application to be successfully unregistered.Callback notification attempts left 0Callback notification trying http://Utility02:11000/oozie/callback?id=0000000-191219193920246-oozie-oozi-W@sqoop-213e&status=FAILEDCallback notification to http://Utility02:11000/oozie/callback?id=0000000-191219193920246-oozie-oozi-W@sqoop-213e&status=FAILED succeededCallback notification succeeded
真的蛋疼啊,我在这个上面浪费了太多的时间
后知后觉
忽然在Hadoop实操的微信公众号里搜了一下Hue,才发现这个公众号早就回答了这个问题,以后多多学习