FusionInsight HD (V100R006)loader 任务报错“The input parameter is null. cause: ftpClient”

发布时间:  2017-06-09 浏览次数:  242 下载次数:  0
问题描述
通过loader从FTP导数据到HDFS,报错“The input parameter is null. cause: ftpClient”,任务日志如下

处理过程
收集详细的job日志,发现有如下报错
2017-06-05 16:16:18,164 INFO [main] org.apache.sqoop.transformation.TransEngine: Not need do transform
2017-06-05 16:16:18,166 INFO [main] org.apache.sqoop.connector.ftp.FtpImportExtractor: map task attempt id: attempt_1496305373805_0005_m_000010_1
2017-06-05 16:16:18,168 INFO [main] org.apache.sqoop.connector.ftp.FtpImportExtractor: Ftp file name: /hbase/data/default/HDS_A1053_NERPF_DTL_JRNLENTR_EXPD_TBL/9e702f43caf13e6e230154315b1e1c13/F/b0618094bd8149659779489982e83989; input path: /hbase/data/default; dest file name: /hbase/data/default/attempt_1496305373805_0005_m_000010_1/HDS_A1053_NERPF_DTL_JRNLENTR_EXPD_TBL/9e702f43caf13e6e230154315b1e1c13/F/b0618094bd8149659779489982e83989, encode:
2017-06-05 16:16:18,185 WARN [main] org.apache.sqoop.core.connector.ftp.util.FtpUtil: Failed to set file transfer mode for ftpClient.
2017-06-05 16:16:18,897 WARN [main] org.apache.sqoop.core.connector.ftp.util.FtpUtil: Connect or Login ftp server [128.196.117.140:22021] failed
2017-06-05 16:16:18,906 INFO [main] org.apache.sqoop.language.Language: Load the path of the language configure : languageConfig
2017-06-05 16:16:18,906 INFO [main] org.apache.sqoop.language.Language: Load language configure : en
2017-06-05 16:16:18,906 INFO [main] org.apache.sqoop.language.Language: Load language configure : zh
2017-06-05 16:16:18,907 INFO [main] org.apache.sqoop.language.LanguageErrorService: Begin to initialize  LanguageError server
2017-06-05 16:16:18,914 INFO [main] org.apache.sqoop.language.Language: Load the path of the language configure : languageErrorConfig
2017-06-05 16:16:18,914 INFO [main] org.apache.sqoop.language.LanguageErrorService: initialize  LanguageError server OK!
2017-06-05 16:16:18,917 INFO [main] org.apache.sqoop.connector.ftp.FtpImportExtractor: Import data from ftp Finish.
2017-06-05 16:16:18,917 INFO [main] org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
2017-06-05 16:16:18,918 INFO [main] org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2017-06-05 16:16:18,918 INFO [main] org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor::SqoopRecordWriter free acquired
2017-06-05 16:16:18,918 INFO [main] org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor::SqoopRecordWriter filled released
2017-06-05 16:16:18,918 INFO [OutputFormatLoader-consumer] org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Loader has finished
2017-06-05 16:16:18,918 INFO [main] org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2017-06-05 16:16:18,919 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.sqoop.common.SqoopException: Error occurs during extractor run.
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:138)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:180)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:174)
Caused by: org.apache.sqoop.common.SqoopException: The input parameter is null. cause: ftpClient
at org.apache.sqoop.utils.Preconditions.checkNotNull(Preconditions.java:28)
at org.apache.sqoop.core.connector.ftp.util.FtpUtil.finalizeFileTransfer(FtpUtil.java:770)
at org.apache.sqoop.connector.ftp.FtpImportExtractor.writeBinaryFile(FtpImportExtractor.java:397)
at org.apache.sqoop.connector.ftp.FtpImportExtractor.processOneSplit(FtpImportExtractor.java:355)
at org.apache.sqoop.connector.ftp.FtpImportExtractor.extractBinaryFile(FtpImportExtractor.java:324)
at org.apache.sqoop.connector.ftp.FtpImportExtractor.extract(FtpImportExtractor.java:192)
日志显示“Connect or Login ftp server [128.196.117.140:22021] failed
进行如下排查

1. 首先确保在loader页面连接器测试连接那块点击test是通过的即用户名密码以及loader节点与ftp服务器网络通 
2. 在运行map失败的节点如下图中的节点ping ftp服务器ip是否可以联通,telnet ftp服务器ip 22021端口是否可以访问 
3. 看到该任务是启动了21个map,能否确认ftp服务器支持的连接数,因为一个map占用一个ftp链接,如果map数大于ftp对外提供的连接数也会出现无法连接、或者其他任务在连接ftp 

最后发现是客户FTP-SERVER设置的连接数为16,将map的数量调成15之后问题解决
根因

因为一个map占用一个ftp链接,如果map数大于ftp对外提供的连接数会出现无法连接

解决方案

map数超过了ftp连接数导致loader任务失败,减小map数之后问题解决

END